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ABSTRACT 


The  Navy  spends  over  $300  million  per  year  to  recruit  approximately 
35,000  new  active  duty  enlisted  Sailors.  The  Navy  has  historically  used  a  non¬ 
linear  optimization  model,  the  Planned  Resource  Optimization  (PRO)  model,  to 
help  inform  decisions  on  the  allocation  of  those  recruiting  resources.  Input 
variables  to  the  PRO  model  include  economic  influences  and  policy  factors.  The 
result  is  a  recommended  allocation  of  resources  for  advertisements,  recruiters, 
enlistment  bonuses,  and  education  incentives.  The  PRO  model’s  primary 
limitations  are  (1)  potential  deviations  of  input  variables  are  not  taken  into 
consideration,  and  (2)  extensive  experimentation  is  not  feasible.  Realistically, 
input  variables  to  the  PRO  model  fluctuate,  are  unpredictable,  and  can  interact 
with  other  variables  to  influence  the  recruiting  environment  and  affect  the  optimal 
allocation  of  recruiting  resources.  This  paper  describes  the  “Planned  Resource 
Optimization  Model  with  Experimental  Design”  (PROM-WED),  a  tool  that 
alleviates  the  limitations  and  enhances  the  analytic  utility  of  the  legacy  PRO 
model.  PROM-WED  embeds  the  legacy  PRO  model  within  a  data  farming 
environment.  PROM-WED’s  graphical  user  interface  and  decision  support 
capability  provide  decision  makers  with  robust  insights  into  variable  interactions 
and  uncertainties  to  better  inform  their  recruiting  resourcing  decisions. 
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EXECUTIVE  SUMMARY 


The  mission  of  the  United  States  Navy  is  “to  maintain,  train  and  equip 
combat-ready  Naval  forces  capable  of  winning  wars,  deterring  aggression  and 
maintaining  freedom  of  the  seas”  (United  States  Navy,  n.d.-b).  Congress 
authorizes  the  Navy  to  maintain  a  force-strength  of  over  300,000  active  duty 
personnel  to  execute  this  mission  (Government  Accountability  Office,  2016,  p.  5). 
The  Navy  spends  over  $300  million  to  recruit  approximately  35,000  new  active 
duty  enlisted  Sailors  each  year  to  sustain  its  manning  strength  (Department  of 
the  Navy,  2015,  p.  7).  Under  the  Deputy  Chief  of  Naval  Operations  for 
Manpower,  Personnel,  Training  and  Education  (N1/MPT&E),  analysts  use 
mathematical  models  to  support  decision  makers  on  personnel  and  budget 
related  resourcing  issues  (United  States  Navy,  n.d.-a).  One  model  that  N1  has 
historically  used  to  inform  recruiting  resourcing  decisions  is  the  Planned 
Resource  Optimization  (PRO)  model. 

The  PRO  model  is  a  deterministic  non-linear  optimization  model  that 
provides  users  with  a  recommended  set  of  resources  to  minimize  the  cost  of 
Navy  recruiting  (Green  &  Mavor,  1994).  The  PRO  model  optimizes  resources 
allocated  to  advertisements,  enlistment  bonuses,  education  incentives,  and 
recruiters.  Input  variables  to  the  model  include  economic  influences  such  as 
unemployment  rate  and  policy  factors  such  as  percentage  of  high  quality  recruits 
(Navy  Recruiting  Command,  2007).  Limitations  of  the  PRO  model  are  (1)  potential 
deviations  of  input  variables  are  not  taken  into  consideration,  and  (2)  extensive 
experimentation  is  not  feasible.  Realistically,  the  input  variables  to  the  PRO 
model  fluctuate,  are  unpredictable,  and  can  interact  with  other  variables  to 
influence  the  recruiting  environment  and  affect  the  optimal  allocation  of  recruiting 
resources. 

To  alleviate  the  limitations  and  enhance  the  analytic  utility  of  the  legacy 
PRO  model,  we  developed  the  “Planned  Resource  Optimization  Model  with 
Experimental  Design”  (PROM-WED).  PROM-WED  embeds  the  legacy  PRO 

xvii 


model  within  a  data  farming  environment.  The  foundation  of  PROM-WED’s  data 
farming  wrapper  is  the  nearly  orthogonal  Latin  hypercube  (NOLH).  The  NOLH 
design  of  experiments  (DOE)  builds  experimental  designs  that  efficiently  and 
effectively  explore  the  solution  space  (Cioppa  &  Lucas,  2007).  This  good  space¬ 
filling  capability  means  that  uncertainties  and  fluctuations  in  input  variables  along 
with  multivariable  interactions  can  be  adequately  investigated  (Sanchez  &  Wan, 
2015). 

The  33  and  129  design  point  NOLH  designs  were  used  to  construct 
PROM-WED’s  data  farming  wrapper.  The  33-point  NOLH  DOE  tests  each 
variable  at  33  levels  and  grows  data  for  33  legacy  PRO  model  runs,  whereas  the 
129-point  NOLH  DOE  tests  each  variable  at  129  levels  and  grows  data  for  129 
legacy  PRO  model  runs.  PROM-WED’s  graphical  user  interface  (GUI)  allows 
users  to  easily  input  a  range  of  values  for  each  input  variable  into  the  NOLH  DOE 
worksheet,  without  need  for  knowledge  or  familiarity  with  data  farming  or  DOE 
techniques  (Sanchez,  2011). 

A  completed  PROM-WED  excursion  grows  a  data  set  for  either  33  or  129 
data  points.  Automatically  generated  sensitivity  analysis  provides  users  with  a 
basic  risk  assessment  picture  focused  on  the  decision  variables  using  the  data 
grown  by  PROM-WED.  Further  insights  into  variable  interactions  and  effects  of 
input  variables  can  be  easily  explored  using  available  data  analysis  software. 
PROM-WED  transforms  the  legacy  PRO  model  into  a  resource  that  N1  can  use 
to  gain  robust  insights  into  the  optimal  allocation  of  recruiting  resources. 

A  scenario  of  interest  to  N1  was  run  and  analyzed  using  PROM-WED. 
Insights  gained  include: 

1.  To  optimize  the  allocation  of  recruiting  resources  in  fiscal  year  2020,  it 
is  recommended  that  less  funds  be  allocated  to  recruiters  and  more 
funds  be  allocated  to  enlistment  bonuses  and  advertisements. 

2.  Advertising  is  the  most  influential  decision  variable.  Over  80  percent  of 
the  total  cost  of  recruiting  variance  is  explained  by  changes  in  the 
recommended  allocation  of  resources  to  advertising. 


3.  Once  relative  pay  exceeds  approximately  1 .00,  changes  in  the  new 
accession  mission  have  little  to  no  effect  on  the  recommended  amount 
of  resources  allocated  to  advertising. 
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I.  INTRODUCTION 

The  mission  of  the  United  States  Navy  is  “to  maintain,  train  and  equip 
combat-ready  Naval  forces  capable  of  winning  wars,  deterring  aggression  and 
maintaining  freedom  of  the  seas”  (United  States  Navy,  n.d.-b).  Under  Title  10  of 
the  United  States  Code,  Congress  authorizes  the  Navy  to  maintain  a  force- 
strength  of  over  300,000  active  duty  personnel  to  execute  this  mission 
(Government  Accountability  Office  [GAO],  2016,  p.  5).  Each  year,  the  Navy 
recruits  approximately  35,000  new  active  duty  enlisted  Sailors  to  sustain  this 
manning  strength  (Department  of  the  Navy,  2015,  p.  7).  The  Deputy  Chief  of 
Naval  Operations  for  Manpower,  Personnel,  Training  and  Education  (MPT&E/N1) 
is  delegated  with  the  responsibility  over  all  Navy  manpower  readiness  matters,  to 
include  recruiting  (United  States  Navy,  n.d.-a). 

Analysts  at  N1  use  mathematical  models  and  simulations  to  support 
decision  makers  in  the  MPT&E  domain  during  the  Planning,  Programming, 
Budget  and  Execution  (PPBE)  process.  The  PPBE  process  is  the  Department  of 
Defense’s  (DOD)  “primary  resource  management  system...  for  all  appropriated 
funding”  (Tomasini,  n.d.).  The  DOD’s  strategy,  force  structure,  and  allocation  of 
resources  are  all  delineated  within  the  annual  PPBE  process  (Tomasini,  n.d.). 
Each  year  during  the  programming  phase,  N1  submits  Program  Objective 
Memorandum  (POM)  inputs,  recommending  how  funds  should  be  allocated 
within  the  Navy’s  MPT&E  domain  (Defense  Acquisition  University  [DAU],  2013, 
p.  4). 

N1  has  historically  used  the  Planned  Resource  Optimization  (PRO)  model 
to  inform  decisions  regarding  the  allocation  of  recruiting  resources  and  estimate 
total  recruiting  costs.  The  PRO  model  is  a  deterministic,  non-linear  optimization 
model  that  provides  users  with  a  recommended  set  of  resources  that  minimizes 
the  cost  of  recruiting  in  order  to  achieve  a  given  recruiting  mission.  The  PRO 
model  can  also  be  used  to  estimate  recruiting  capacity  for  a  given  level  of 
resources. 
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The  PRO  model  is  built  in  Microsoft  Excel  using  both  worksheet  functions 
and  Visual  Basic  for  Applications  (VBA)  code.  The  PRO  model’s  primary  function 
is  to  provide  a  broad,  estimated  budget  picture  of  Navy  recruiting  resource 
allocation  in  support  of  the  POM.  The  PRO  model  is  also  used  to  answer 
questions  such  as,  “what  is  the  least  expensive  way  to  meet  a  recruiting 
mission?”  and  “how  much  money  do  we  need  to  allocate  for  advertising  to  meet  a 
given  accession  mission?”  (Hogarth,  Lucas,  &  McLemore,  2016,  p.  3576) 

A.  PROBLEM  STATEMENT 

Recruiting  a  high  quality  all-volunteer  force  (AVF)  is  expensive.  The  Navy 
requires  a  growing  number  of  high  quality  recruits  to  meet  the  needs  of  its 
technologically  advanced  fleet.  In  a  perfect  world,  N1  would  be  given  a  blank 
check  to  cover  the  cost  of  recruiting  a  100  percent  high  quality  force.  However,  in 
reality,  N1  faces  a  fiscally  constrained  environment. 

The  PRO  model  is  a  deterministic,  non-linear  optimization  model  that 
provides  users  with  a  recommended  set  of  resources  that  attempts  to  minimize 
the  cost  of  recruiting  (Green  &  Mavor,  1994).  Analysts  at  N1  use  the  PRO  model 
to  optimize  resources  allocated  to  advertisements,  enlistment  bonuses, 
education  incentives,  and  recruiters.  Input  variables  include  economic  influences 
such  as  unemployment  rate,  and  policy  factors  such  as  target  percentage  of  high 
quality  recruits  (Navy  Recruiting  Command,  2007).  The  PRO  model’s  primary 
limitations  are  (1)  deviations  of  input  variables  are  not  taken  into  consideration, 
and  (2)  extensive  experimentation  capability  is  not  available.  Realistically,  the 
input  variables  to  the  PRO  model  fluctuate,  are  unpredictable,  and  can  interact 
with  other  variables  to  influence  the  recruiting  environment  and  affect  the  optimal 
allocation  of  recruiting  resources. 

B.  THESIS  PURPOSE 

The  objective  of  this  research  is  to  develop  a  tool  that  transforms  the  PRO 

model  into  a  tool  that  provides  N1  analysts  with  a  robust  decision  support 

capability  for  recruiting  resourcing  decisions.  In  this  research,  the  author  wrapped 

2 


a  design  of  experiments  (DOE)  capability  around  the  legacy  PRO  model.  We  call 
the  enhanced  tool  the  Planned  Resource  Optimization  Model  with  Experimental 
Design  (PROM-WED). 

The  data  farming  wrapper  in  PROM-WED  provides  legacy  PRO  model 
users  with  the  ability  to  input  a  range  of  possible  values  for  input  and  decision 
variables.  The  legacy  PRO  model  is  run  over  each  scenario  that  is  formulated  by 
the  DOE  tool.  Instead  of  a  single,  discrete  solution  found  by  the  legacy  PRO 
model,  PROM-WED  grows  data  that  gives  robust  insight  into  cause  and  effect 
relationships  amongst  the  variables. 

C.  RESEARCH  QUESTIONS 

In  order  to  best  provide  N1  analysts  with  a  tool  that  improves  their 
decision  support  analysis  for  recruiting  budget  estimates  and  resource  allocation, 
this  research  is  guided  by  the  following  questions: 

1.  How  can  design  of  experiment  techniques  better  inform  decision 
maker’s  determination  of  the  optimal  and  robust  combination  of 
recruiting  resources? 

2.  How  can  efficient  design  of  experiment  techniques  be  incorporated 
around  the  PRO  model  for  future,  on-the-spot  risk  and  sensitivity 
analysis? 

3.  Can  an  enhanced  PRO  model  give  decision-makers  a  robust 
solution  for  the  optimal  allocation  of  recruiting  resources? 

D.  METHODOLOGY 

PROM-WED’s  data  farming  wrapper  uses  the  Nearly  Orthogonal  Latin 
Hypercube  (NOLH)  DOE  worksheet  tool  developed  by  the  Simulation 
Experiments  &  Efficient  Designs  (SEED)  Center  for  Data  Farming  at  the  Naval 
Postgraduate  School  (NPS),  see  https://harvest.nps.edu.  A  new  graphical  user 
interface  (GUI)  allows  the  user  to  input  a  range  of  values  for  each  input  variable 
into  the  NOLH  DOE  worksheet.  The  NOLH  DOE  worksheet  was  embedded  into 
the  PRO  model.  The  user  has  the  option  to  run  an  excursion  using  a  33-point 
design  or  a  129-point  design.  PROM-WED  generates  a  robust  recommended 
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allocation  of  recruiting  resources.  Basic  sensitivity  analysis  provides  the  user  with 
a  risk  assessment  picture,  and  further  analysis  can  be  completed  using  any  data 
analysis  software  package,  such  as  JMP.  Scenarios  of  interest  to  N1  are  run  and 
analyzed. 

E.  BENEFITS  OF  RESEARCH 

The  ability  to  quickly  explore  scenarios  with  a  deterministic  optimization 
model  using  efficient  DOE  techniques  provides  N1  with  richer  insights  into 
combinations  of  resources  that  can  be  utilized  to  achieve  a  given  active  enlisted 
recruiting  mission.  Instead  of  a  discrete  expected  value,  the  implementation  of 
efficient  DOE  techniques  provides  decision-makers  with  a  “robust  [foundation  to 
make]  decisions  or  policies”  (Sanchez,  Sanchez,  &  Wan,  2014,  p.  1).  DOE 
methods  will  also  provide  improved  insight  into  tradeoff  relationships  between 
input  parameters  and  the  output  results  (Vieira,  Sanchez,  Kienitz,  &  Belderrain, 
2013,  p.  264).  N1  will  also  benefit  from  this  study  by  gaining  a  tool  that  provides 
on-the-spot  sensitivity  analysis  using  sophisticated  DOE  techniques. 

F.  ORGANIZATION  OF  THESIS 

This  thesis  is  comprised  of  five  chapters.  Chapter  I  focuses  on  the 
motivation  of  the  thesis  and  explains  how  the  research  questions  are  addressed. 
Chapter  II  discusses  the  history  and  composition  of  the  PRO  model  and 
considers  other  research  that  has  been  done  on  military  recruiting  resource 
allocation  and  the  implementation  of  data  farming  on  simulation  models.  Chapter 
III  addresses  the  methodology  used  to  build  PROM-WED,  including  a  review  of 
DOE  techniques,  and  the  components  of  PROM-WED.  Chapter  IV  introduces 
scenarios  of  interest  to  N1,  and  the  remainder  of  the  chapter  provides  an 
analysis  of  the  data  generated  for  these  scenarios  using  PROM-WED.  Last, 
Chapter  V  provides  concluding  remarks,  and  recommendations  for  further  work. 


4 


II.  BACKGROUND 


In  this  chapter,  research  on  military  recruiting  resource  allocation  is 
presented,  followed  by  a  conceptual  overview  of  the  PRO  model. 

A.  LITERATURE  REVIEW 

All  military  branches  face  the  challenge  of  determining  the  best  way  to 
allocate  recruiting  resources.  PRO  is  a  model  that  the  Navy  has  historically  used 
to  help  decision  makers  gain  insight  to  answer  this  question. 

1.  Recruiting  Resource  Allocation  Models 

Following  the  United  States  withdrawal  from  the  Vietnam  War  in  1973, 
Congress  terminated  conscription,  and  the  military  transitioned  to  an  All- 
Volunteer  Force  (AVF)  (Morey  &  McCann,  1 980,  p.  1198).  Critics  of  an  AVF  were 
concerned  that  it  would  result  in  weakened  national  security  due  to  low  quality 
recruits  and  insufficient  accession  numbers.  In  contrast  to  a  military  manned  by 
conscripts,  an  AVF  forced  each  service  to  expend  more  effort  “to  meet  the 
various  quantity  and  quality  goals”  for  recruiting  new  enlistees  (Morey  &  McCann, 
1980,  p.  1198).  Various  modeling  efforts  were  made  to  gain  insight  into  how  to 
best  allocate  recruiting  resources  to  meet  the  service’s  set  recruiting  goals. 

In  1978,  Chappell  and  Peel  developed  static  and  dynamic  optimization 
models  to  determine  the  optimal  allocation  of  advertising  resources  to  achieve 
military  recruiting  goals.  The  dynamic  model  they  developed  introduced 
economic  factors  such  as  labor  supply  and  incorporated  current  and  past 
recruiting  data  to  determine  an  optimal  allocation  of  advertising  resources 
(Chappell  &  Peel,  1978,  p.  910). 

In  1980,  Morey  and  McCann  developed  a  model  to  determine  the  optimal 
allocation  of  recruiting  resources  by  inputting  econometric  data  of  a  given  region 
and  descriptive  data  that  reflects  its  demographic  population.  The  model  was 
conducive  to  “performing]  sensitivity  analyses  related  to  the  impacts”  of  various 
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economic  and  demographic  changes.  Their  model  identified  the  percentage  of 
recruits  who  graduated  from  high  school  as  the  indicator  for  recruit  quality  (Morey 
&  McCann,  1980,  p.  1204). 

2.  Cost-Performance  Tradeoff  Model 

AVF  concerns  peaked  in  1980,  particularly  in  regards  to  how  the  military 
gauged  recruit  quality  (Green  &  Mavor,  1994,  p.  8).  The  DOD  informed  Congress 
that  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB),  the  examination 
used  to  determine  enlistment  eligibility,  was  incorrectly  scored  between  1976  and 
1980  (Green  &  Mavor,  1994,  p.  2).  This  error  resulted  in  hundreds  of  thousands 
of  people  entering  military  service  who  did  not  meet  enlistment  standards  (Green 
&  Mavor,  1994,  p.  2). 

Clinical  psychologists  advised  that  the  recruits  who  did  not  meet  minimum 
enlistment  standards  were  classified  as  individuals  who  “generally  need  intense 
supervision  and  guidance,  particularly  under  conditions  of  serious  stress” 
(Laurence  &  Ramsberger,  1991,  p.  8).  These  attributes  are  undesirable  for 
military  service. 

In  reaction  to  this  mistake,  Congress  tasked  the  DOD  “to  link  enlistment 
standards  to  job  performance”  (Green  &  Mavor,  1994,  p.  2).  This  initiated  the 
Joint-Service  Job  Performance  Measurement/Enlistment  Standards  (JPM) 
Project.  The  first  phase  of  the  JPM  project  “concentrated  on  developing  a  variety 
of  measures  of  job  performance  so  that  enlistment  standards  could  be  related  to 
something  close  to  actual  performance  on  the  job”  (Green  &  Mavor,  1994,  p.  7). 

Following  decades  of  research,  phase  one  of  the  JPM  project  validated 
the  ASVAB  as  “a  reasonably  valid  predictor  for  performance  in  entry-level  military 
jobs”  (Green  &  Mavor,  1994,  p.  10).  High  school  graduation  became  an  indicator 
of  likelihood  of  first  term  enlistment  completion.  The  military  services  now  faced 
the  challenge  of  determining  “how  much  quality  can  we  afford?”  since  “high- 
quality  personnel  cost  more  to  recruit,  and  the  public  purse  is  not  bottomless” 
(Green  &  Mavor,  1994,  p.  4,10-11).  The  goal  of  the  second  phase  of  the  JPM 
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project  was  to  address  this  question.  The  cost-performance  tradeoff  (CPT)  model 
was  their  solution  (Green  &  Mavor,  1 994,  p.  11). 

The  CPT  model  is  a  tool  that  decision  makers  use  to  estimate  the 
“probable  effects  on  performance  and/or  costs  of  various  scenarios”  (Green  & 
Mavor,  1994,  p.  11).  The  CPT  model  is  comprised  of  “four  primary  components: 
(1)  the  performance  equations,  (2)  the  recruiting  cost  function,  (3)  survival  rates, 
and  (4)  training  and  compensation  costs”  (McCloy  at  al.,1992,  p.  iii).  The  PRO 
model  is  based  upon  the  recruiting  cost  function,  which  is  covered  in  the  next 
section. 

B.  PLANNED  RESOURCE  OPTIMIZATION  MODEL 

The  PRO  model  is  a  non-linear  optimization  model  implemented  in 
Microsoft  Excel  using  both  worksheet  functions  and  VBA  code.  It  evaluates  user 
driven  input  variables  over  a  recruiting  cost  function.  The  result  is  a 
recommended  combination  of  recruiting  resources  to  meet  a  given  recruiting 
mission. 

1.  Components  of  the  PRO  Model 

The  PRO  model  uses  the  recruiting  cost  function  from  the  CPT  model  to 
allocate  recruiting  resources  while  minimizing  the  cost  of  recruiting.  A  general 
review  of  the  PRO  model’s  conceptual  framework  and  an  overview  of  the 
workings  of  the  PRO  model  follow. 

a.  Conceptual  Framework 

Input  variables  to  the  PRO  model  include  decision  variables,  market 
factors,  and  policy  factors  that  affect  the  cost  and  nature  of  recruiting.  Users  can 
change  these  inputs  to  test  different  recruiting  scenarios.  Figure  1  shows  a 
conceptual  representation  of  the  PRO  model. 
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Figure  1.  Conceptual  Representation  of  the  PRO  Model 


Inputs 


•  Recruiting  Mission 

*  Percentage  of  high 
quality  recruits 

*  Total  accession  mission 

•  Market  Environmental  Data 

*  Unemployment  data 

*  Relative  pay  data 

*  Qualified  Military 
Available  pool 


PRO  Model  Solving 
Recruiting  Cost  Function 


Adapted  from  Navy  Recruiting  Command  (2007,  p.  5). 


Outputs 

•  Number  and  Cost  of 
Recruiters 

•  Advertising  Dollars 

•  Enlistment  Bonus 

•  Education  Benefits 


A  PRO  model  excursion  produces  a  point  solution  that  tells  the  user  how 
many  production  recruiters  should  be  in  the  field  along  with  the  allocation  of 
funds  towards  enlistment  bonuses,  education  incentives,  and  advertisements.  An 
example  output  of  a  PRO  model  excursion  is  shown  in  Figure  2. 


Figure  2.  PRO  Model  Output 


Resource  Run 

2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCO 

35.025 

36.425 

36.800 

35,800 

35.225 

34,650 

34.650 

Capacity 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Unemployment  (%) 

40 

40 

40 

40 

40 

40 

40 

Float 

Total  Recruiters 

4,092 

4,632 

4  608 

4,315 

4,106 

3,945 

3,923 

Total  Recruiter  Cost  (SM) 

S335  138 

$383  495 

$387  265 

$368  083 

S355  535 

S347 480 

$351  645 

Float 

Advertising  (SM) 

$86  203 

$105  531 

$105  352 

$97  179 

$91  278 

$87  362 

$87  981 

Fixed 

Enlistment  Bonus  (SM) 

S40971 

S36  580 

$41,340 

$40  650 

$42  230 

$42060 

$42  810 

Float 

Education  Incentives  (SM) 

SO  000 

$0  000 

SO  000 

SO  000 

SO  000 

$0  000 

$0  000 

LRP  (SM) 

$7  440 

S1 1  220 

$11  280 

$11  380 

$11430 

S1 1460 

S1 1  670 

HSDG 

95% 

95% 

95% 

95% 

95% 

95% 

95% 

TSC  l-IIIA 

70% 

70% 

70% 

70% 

70% 

70% 

70% 

Total  Cost  (SM) 

$469  762 

$536  826 

$545  236 

$517292 

S500  473 

$488  361 

$494  106 

Program  Objective  Memorandum  (POM)  FY17  version  of  the  PRO  model. 


The  interpretation  of  the  point  solution  for  fiscal  year  (FY)  2015  in  Figure  2 
is:  To  meet  an  accession  mission  of  35,025  new  Navy  recruits  with  a  fixed 
enlistment  bonus  budget  of  $40,971 ,000,  the  Navy  should  allocate  4,092  Sailors 
to  recruiting  duty  and  $86,203,000  to  advertising. 
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b.  The  Recruiting  Cost  Function 

The  recruiting  cost  function  is  “the  underpinning  of  the  [PRO]  model” 
(Navy  Recruiting  Command,  2007,  p.  6).  The  black  box  shown  previously  in 
Figure  1  represents  the  recruiting  cost  function.  Users  cannot  alter  the  recruiting 
cost  function  to  include  the  elasticities  or  pre-set  data  that  feed  into  it. 

The  recruiting  cost  function  provides  the  “minimum  cost  budget” 
recommended  to  recruit  “a  specified  number  of  individuals”  while  taking  into 
consideration  the  conditions  of  the  “recruiting  market”  (Green  &  Mavor,  1994,  p. 
126-127).  Enlistment  supply  functions  and  a  constrained  minimization  problem 
are  both  critical  components  of  the  recruiting  cost  function  (Green  &  Mavor, 
1994,  p.  126-127). 

The  variables  of  interest  that  build  the  recruiting  cost  function  include 
(Green  &  Mavor,  1994,  p.  126-127;  Katznelson,  2010,  p.  4;  Navy  Recruiting 
Command,  2007,  p.  8.): 

H  =  High  quality  recruits 
M  =  Medium  quality  recruits 
L  =  Low  quality  recruits 

Cq  =  Number  of  contracts  signed  in  a  given  year,  where  q  =  H,M,L 
Cq  =  Constant  ( calculated  using  base  year  knowns ),  where  q  =  H,M,L 
C?  =  Minimum  cost  to  contract  given  number  of  recruits,  where  q  =  H,M,L 
Xq  =  Lagrangian  multiplier,  where  q  =  H,M,L 

a  =  Elasticity  describing  the  relationship  between  the  paramter  and  CH 
Rq  =  Number  of  production  7'eci'uiters  to  recruit  a  given  quality,  where  q  —  H,M,L 
AD  =  Advertising  dollars  spent  in  the  model  year  ( inflation  —  adjusted  to  base  year) 
B  =  Average  enlistment  bonus  paid  to  q  =  H  ( model  year) 

E  =  Average  education  benefits  paid  to  q  =  H  ( model  year) 
v  =  Price  mdex  to  deflate  B  and  E  into  base  year  dollars 

F  =  Factors  affecting  the  recruiting  market  (i.  e.  .unemployment,  relative  pay) 

T  =  Testing  cost| 

O  =  Fixed  costs  of  recruiting 

The  enlistment  supply  functions,  shown  in  Figure  3,  are  separated  into 

high,  medium,  and  low  quality  categories  of  new  accessions.  These  equations 

determine  the  expected  number  of  recruits  in  each  category  that  will  be 

contracted  per  year  (Green  &  Mavor,  1994,  p.  126). 
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Figure  3.  Enlistment  Supply  Functions 

High  Quality  Contracts : 

ln(CH)  =  ln(C^)  +  aF  In (RH)  +  aAD  In (FH)  +  aBln  ^  +  aEln  j  +  aF  ln(F) 

Medium  Quality  Contracts : 

ln(CM)  =  In(C")  +  a%  In (FM)  +  aF  ln(F) 

Low  Quality  Contracts: 

In(CL)  =  ln(Cg)  +  In (RL)  +  aF  ln(F) 

Adapted  from  Green  &  Mavor,  1994,  p.  1 26;  Navy  Recruiting  Command  (2007,  p. 

8). 

The  “Navy  Recruiting  Cost  Model  User  Manual”  refers  to  the  enlistment 
supply  functions  as  recruiting  cost  functions  (Navy  Recruiting  Command,  2007, 
p.8).  Smith  and  Hogan  refer  to  these  functions  as  the  enlistment  supply  functions 
in  “Modeling  Cost  and  Performance  for  Military  Enlistment,  Report  of  a 
Workshop”  (Green  &  Mavor,  1994,  p.126).  For  the  purpose  of  this  research,  the 
functions  shown  in  Figure  3  are  referred  to  as  enlistment  supply  functions. 

The  objective  function  shown  in  Figure  4  determines  “the  levels  of 
recruiting  resources  and  incentives  required  to  recruit  the  specified  mission  at 
minimum  cost”  (Green  &  Mavor,  1994,  p.  127). 

Figure  4.  Recruiting  Cost  Minimization  Problem 

Min[pR{RH  +  Rm  +  Rl )  +  pADAD  +  C?(B  +  E)  +  T(C?  +  C "  +  C/*)  +  O ] 

-Me"  +  C"]  -  XM[CM  -  C«]  -  XJC*  -  cl) 

Source:  Green  and  Mavor  (1994,  p.  127). 

An  explanation  of  the  components  that  makeup  the  recruiting  “cost 
minimization  problem”  is  shown  in  Figure  5. 
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Figure  5.  Recruiting  Cost  Minimization  Component  Roadmap 

First  Line:  Recruiting  Budget  Formula 


Cost  of  Maintaining  Cost  of  Cost  of  Recruiting  Cost  of 

Recruiters  Advertising  Incentives  Testing  Recruits  Fixed  Costs 

i - * - 1  r^i  i — * — i  i — ^ - 1 

Min[pR(RH  +  R M  +  Rl )  +  pAD  AD  +  C?  (B  +  E)  +  T(C?  +  C?  +  C.L)  +  O] 

-A h[Ch  +  C4W]  -  A m[Cm  -  C "]  -  A l[Cl  -  CtL] 


Second  Line:  Constraints  to  ensure  desired  recruiting  mission  is  met 

Adapted  from  Green  and  Mavor  (1994,  p.  127). 

The  first  order  conditions  of  the  recruiting  minimization  problem  are  then 
“substituted”  into  the  recruiting  budget  formula  (i.e.,  the  first  line  of  the  recruiting 
cost  minimization  problem  shown  in  Figure  5)  to  “yield  the  recruiting  cost 
function,”  shown  in  Figure  6  (Green  &  Mavor,  127). 


Figure  6.  Recruiting  Cost  Function 


i+gB+gg 
a 


Minimum  Cost  Budget  =  aZ{CR)  «  +  p 

+T(C"  +  +  O')  +  0 

where : 


1  ' 
R 


Z  = 


“  {«AD\—  zEz 

^f)  («B)  “  M  «  (F)  a 


and  a  =  a%  +  aAD  +  aB  +  aE 
Adapted  from  Green  and  Mavor  (1994,  p.  127). 
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Elasticities  within  the  recruiting  cost  function  are  parameters  built  into  the 
model  that  “represent  the  percent  change  in  enlistment  contracts  for  a  percent 
change  in  recruiting  resources/incentives  or  market  factors”  (McCioy  et  ai.,  1992, 
p.  74).  The  PRO  model  elasticities  were  last  revised  when  the  SAG  Corporation 
and  The  Lewin  Group,  Inc.  updated  the  PRO  model  in  September  2011  (Hogan, 
Warner,  &  Mackin,  n.d.).  More  information  about  the  derivation  and  specifics  of 
the  cost  performance  tradeoff  model  can  be  found  in  the  “Job  Performance,  and 
Cost:  A  Cost-Performance  Tradeoff  Model”  report  and  “Modeling  Cost  and 
Performance  for  Military  Enlistment,”  a  report  of  a  workshop  (McCioy  et  al.,  1992; 
Green  &  Mavor,  1994). 

c.  Model  Framework 

The  PRO  model  is  made  up  of  four  worksheets:  (1)  User  Interface,  (2) 
Inputs  (will  be  referred  to  as  the  “Data  Worksheet”  in  this  research),  (3) 
Simulation,  and  (4)  Results.  Figure  7  illustrates  the  conceptual  flow  and 
relationship  between  these  worksheets. 


Figure  7.  Structure  of  the  PRO  Model 


/  > 

User  Interface  Worksheet 

•  Input  values  of  interest 

for  scenario. 

_ / 

Results  Worksheet 

r - — > 

•  Displays  optimal 
allocation  of  recruiting 
—  resources  as  a  point 
solution. 

V. _ y 


Adapted  from  Navy  Recruiting  Command  (2007,  p.  12). 


The  “User  Interface”  worksheet  allows  the  user  to  enter  values  for  each 
input  variable,  select  which  decision  variables  are  fixed  or  to  be  optimized,  and 
select  how  the  model  is  run  over  seven  FYs.  The  “Data  Worksheet”  and  the 
“Simulation  Worksheet”  are  components  of  the  “black  box”  that  makeup  the 
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recruiting  cost  function.  The  “Results  Worksheet”  provides  users  with  the  optimal 
mix  of  recruiting  resources  to  minimize  the  cost  of  recruiting.  Each  component  of 
the  PRO  model  is  now  explained  in  more  detail. 

2.  PRO  Model  Variables 

The  variables  of  the  PRO  model  are  classified  into  three  types:  (1) 
decision  variables,  (2)  market  factors,  and  (3)  policy  factors.  Variables  are  fed 
into  the  model  through  PRO’S  “User  Interface”  Microsoft  Excel  worksheet,  as 
shown  in  Figure  8. 


Figure  8.  The  PRO  Model  User  Interface 


Step  1:  Check  input 

Model  Feeds 

FY15  FY16 

FY17 

FY18  FY19 

FY20 

FY21 

NCF  +  College  First 

0.000  0.000 

0.000 

0.000  0.000 

0.000 

0  000 

Unemployment  Rate 

6.40%  5.10% 

4.90% 

4.80%  4.80% 

4.80% 

4.80% 

Recruiters 

3.913  3,685 

3.685 

3,685  3.685 

3,685 

3.685 

LRP 

7.440  11.220 

11  280 

11.380  11.430 

11.460 

11.670 

Advertising  (AC  Enl.  Only) 

34.826  34.699 

35.679 

36  729  39  886 

38  892 

39  669 

EB 

40.971  36.580 

41.340 

40  650  42230 

42  060 

42.810 

NCO  (50%  BoY  DEP) 

'  35.025  '  36.425 

"  36.800 

"  35.800  r  35.225 

"  34.650 

"  34.650 

Step  2:  Select  switches 

Total  Recruiters 

Float 

Fixed 

Advertising 

Float 

Float 

Bonus 

Fixed 

Education 

Float 

Step  3:  Select  Best/Worst  Cases  for  Unemployment 

POR  +/-  Range 

610% 

Range  (+/-) 

0.50% 

High  UE  Case 

7.40% 

r  5.90% 

r  5.80% 

r  5.80% 

r  5.80% 

r  5.80% 

POR  +/-  Range 

Base  UE  Case 

6.90% 

r  5.60% 

r  5.40% 

r  5.30% 

r  5.30% 

r  5.30% 

r  5.30% 

Manual  Forecast 

Low  UE  Case 

6.40% 

r  5.10% 

r  4.90% 

r  4.80% 

r  4.80% 

r  4.80% 

r  4.80% 

Manual  Forecast 

High  UE 

5.00% 

5.00% 

5.00% 

5.00% 

5.00% 

5.00% 

5.00% 

Consensus  UE 

4.50% 

4.50% 

4.50% 

4.50% 

4.50% 

4.50% 

4.50% 

Low  UE 

4.00% 

4.00% 

4.00% 

4.00% 

4.00% 

4.00% 

4.00% 

Step  4  Select  run  type 

Traditional  Run 

capacity  Kun 

POM  FY17  version  of  the  PRO  model. 


a.  Decision  Variables 

The  decision  variables  of  the  PRO  model  are  the  resources  that  the  Navy 

controls.  N1  can  influence  how  much  is  budgeted  toward  advertising,  how  many 

production  recruiters  are  in  the  field,  and  how  funding  is  allocated  for  enlistment 
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bonuses  and  education  incentives.  It  is  useful  to  know  the  optimal  balance 
between  these  resources,  since  once  money  is  allocated  to  a  certain  resource, 
those  funds  cannot  be  used  for  anything  else.  For  example,  funds  appropriated  for 
recruiters  cannot  be  used  to  fund  advertising  (Morey  &  McCann,  1980,  p.  1198). 

(1)  Advertising 

Advertising  is  defined  by  the  United  States  Government  Accountability 
Office  (GAO)  as  “the  placement  of  messages  intended  to  inform  or  persuade  an 
audience  through  various  types  of  media  such  as  television,  radio,  digital  media, 
direct  mail,  and  others”  (GAO,  2016,  p.  1).  Each  military  service,  including  the 
Navy,  uses  the  advertising  construct  shown  in  Figure  9  to  “raise  the  public’s 
awareness...  and  help  recruiters  generate  leads  of  potential  recruits”  (GAO, 
2016,  p.  1).  A  lead  is  someone  who  shows  interest  in  joining  the  Navy.  Leads  can 
be  generated  a  variety  of  ways,  from  face-to-face  interactions  with  a  recruiter  to 
indirect  contact  through  advertisement  efforts. 


Figure  9.  Phases  and  Goals  of  Military  Advertisement 


Awareness: 

television  commercials,  signs,  banners 


Engagement: 

websites,  digital  banner  advertisments,  social  media 


Lead  Generation: 

direct  mail,  recruiting  booths,  online  or  print  job  postings 


Adapted  from  GAO  (2016,  p.  7). 


Military  advertising  efforts  are  in  line  with  the  “consumer  journey”  construct 
found  in  the  private  sector.  The  goal  is  “to  move  a  potential  recruit  through  each 
phase  and,  ultimately,  to  a  decision  to  enlist”  (GAO,  2016,  p.  7).  The  Navy 
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typically  allocates  about  $50  million  each  FY  towards  advertising.  Table  1  shows 
a  breakdown  of  how  much  the  Navy  has  annually  allotted  for  advertising  over  the 
past  three  FYs  in  comparison  to  the  other  military  services  (GAO,  2016,  p.  41). 


Table  1 .  Reported  Annual  Allotments  for  Military  Advertising 


2015  Actuals 

2016  Enacted 

2017  Estimate 

Navy 

$56,100,000 

$49,000,000 

$47,000,000 

Army 

$367,700,000 

$238,100,000 

$292,600,000 

Air  Force 

$59,400,000 

$35,900,000 

$60,300,000 

Marine  Corps 

$86,300,000 

$81,500,000 

$81,800,000 

These  values  include  active  duty  and  reserve  recruiting  budgets.  Adapted  from 

GAO  (2016,  p.  41). 

(2)  Enlistment  Bonus 

Enlistment  bonuses  are  used  to  incentivize  high  quality  applicants  to  join 
the  Navy.  Enlistment  bonuses  are  tied  to  specific  Navy  occupational  specialties. 
Special  warfare  (Navy  sea,  air,  and  land  (SEAL)  operators)  and  nuclear  field  (NF) 
specialties  are  examples  of  Navy  occupational  specialties  that  require  high 
ASVAB  scores,  and  regularly  offer  an  enlistment  bonus  to  recruits.  Figure  10 
shows  the  individual  enlistment  bonus  offered  to  each  recruit  who  enlisted  as  a 
special  operator  or  in  the  nuclear  field.  Enlistment  bonuses  fluctuate  for  many 
reasons,  to  include:  the  time  of  year  the  recruit  ships  to  boot  camp,  and  under  or 
over  manning  strength  of  the  Navy  occupational  specialty  (Navy  Recruiting 
Command,  n.d.). 
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Figure  10.  Enlistment  Bonus  Offered  to  NF  and  Special  Operator 

Recruits 


Data  gathered  from  Navy  Recruiting  Command  enlistment  bonus  messages  from 
October  2002  to  February  2016.  Adapted  from  Navy  Recruiting  Command  (n.d.). 


(3)  Production  Recruiters 

Production  recruiters  are  Navy  Sailors  assigned  to  recruiting  duty  with  the 
9585  Navy  Enlisted  Classification  (NEC)  code.  For  model  simplicity,  the  PRO 
model  divides  production  recruiters  into  three  quality  categories:  low,  medium,  and 
high.  The  model  designates  high  quality  recruiters  to  recruit  only  high  quality 
applicants,  and  so  on.  Navy  Recruiting  Command  (NRC)  does  not  separate 
recruiters  into  these  tiered  categories.  This  simplifying  assumption  helps  represent 
that  “high-quality  personnel  cost  more  to  recruit”  (Green  &  Mavor,  1994,  p.  10-11). 

The  number  of  production  recruiters  in  the  field  directly  affects  the  total 
cost  of  recruiting.  The  PRO  model  takes  recruiter’s  base  pay  and  individual 
support  costs  into  consideration  when  calculating  the  cost  of  recruiting. 

(4)  Education  Incentive 

Following  the  enactment  of  the  Post  9/1 1  Gl  Bill  in  June  of  2008,  the  Navy 
has  not  allocated  funds  towards  the  legacy  Navy  college  fund  (NCF)  for  new 
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recruits  (Dortch,  2014,  p.  1;  Palmer,  personal  communication,  June  2016).  The 
Post  9/11  Gl  Bill  provides  service  members  who  have  served  on  active  duty 
following  September  10,  2001,  with  education  benefits  that  “can  cover  all  in-state 
tuition  and  fees  at  public  degree  granting  schools”  along  with  support  programs 
for  out-of-state  and  private  institutions  (Department  of  Veterans  Affairs,  2012). 

b.  Market  Factors  of  the  PRO  Model 

Market  factors  of  the  PRO  model  are  economically  and  demographically 
driven  variables  that  are  uncontrollable  and  subject  to  unexpected  change. 

(1)  Unemployment  Rate 

Unemployment  rates  are  shown  to  be  “positively  and  significantly  related 
to  high-quality  enlistment  contracts”  (Asch  et  al.,  2010,  p.  21).  As  shown  in  Figure 
8,  this  trend  indicates  that  higher  unemployment  rates  lead  to  more  high  quality 
enlistment  contracts  (Bicksler  &  Nolan,  2009,  p.  5).  Figure  1 1  shows  data 
aggregated  for  all  military  services. 

Figure  11.  Unemployment  and  High-quality  Enlistments 


Source:  Bicksler  and  Nolan  (2009,  p.  5). 
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(2)  Relative  Pay 

Like  unemployment  rates,  military  recruiting  for  high  quality  applicants 
responds  “to  the  level  of  military  pay  relative  to  civilian  sector  wage  opportunities” 
(Warner,  2012,  p.  71).  The  PRO  model  captures  this  market  driver  through  the 
relative  pay  ratio.  Since  the  Navy  requires  high  quality  Sailors  who  are  technically 
competent,  “pay  comparability...  is  an  issue  for  certain  hard-to-fill  occupations 
and  skills  that  command  high  salaries  in  the  civilian  sector,  particularly  in  high 
technology  fields”  (Bicksler  &  Nolan,  2009,  p.  34).  Table  2  provides 
interpretations  of  relative  pay  ratios. 


Table  2.  Interpretations  of  Relative  Pay 


Relative  Pay 

Interpretation 

0.5 

Military  pay  is  50  cents  to  every  dollar  of  civilian  sector  pay. 

1.0 

Military  pay  is  equal  to  civilian  sector  pay. 

2.0 

Civilian  sector  pay  is  50  cents  to  every  dollar  of  military  pay. 

(3)  Qualified  Military  Available 

Qualified  military  available  (QMA)  is  an  estimate  of  the  “17-  to  24-year-old 
youth  population  in  the  United  States  who  would  qualify  without  needing  a  waiver 
and  be  available  to  enlist  in  the  active  component  military”  (Office  of  the 
Undersecretary  of  Defense,  2016,  p.  2).  An  independent  firm,  Woods  and  Poole 
Economics,  provides  the  Navy  with  QMA  data  (www.woodsandpoole.com). 

Some  common  disqualifiers  for  applicants  joining  the  military  include:  illicit 
drug  use,  overweight/obesity,  use  of  prescribed  psychotropic  drugs,  and  failure  to 
complete  high  school.  Figure  12  is  a  hypothetical  model  which  depicts  the  QMA 
pool  for  the  recruiting  efforts  of  the  four  military  services.  The  resulting  pool  of 
QMA  is  just  a  small  portion  of  the  overall  military-aged  population  within  the 
United  States. 
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Figure  12.  Hypothetical  Breakdown  for  Estimated  QMA  Pool  in  2030 


In  2030  there  will  be  approximately  30  million  military 
aged  (18  -  24)  people  in  the  United  States 


Data  approximations  are  adapted  from  MarketingCharts  (2016);  Child  Trends  Data 
Bank  (2014);  Child  Trends  Data  Bank  (2016);  National  Center  for  Education 
Statistics  (2016).  Propensity  to  serve  metric  is  omitted  due  to  distribution 
restrictions. 


Other  factors  that  are  not  considered  include  the  percentage  of  young 
adults  who  are  currently  enrolled  in  college,  those  who  are  permanently 
employed,  or  those  who  may  have  dependents. 

c.  Policy  Factors  of  the  PRO  Model 

Policy  factors  are  variables  that  can  be  adjusted,  but  are  done  through  a 
combination  of  policy,  service  culture,  and  budget  changes. 

(1 )  New  Contract  Objective 

The  new  contract  objective  (NCO)  is  the  Navy’s  enlisted  accession 
mission  for  a  given  FY.  NRC’s  NCO  goal  each  FY  is  dependent  on  the  Navy’s 
projected  end-strength.  The  equation  for  end-strength  is  shown  in  Figure  13. 
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Figure  13.  Equation  for  Navy  Enlisted  Strength  Planning 


f 

Fhxibh 

- - 4 

-  Mandated  — H 

Begin 

Losses 

+ 

Gains 

— 

End 

Strength 

Strength 

•  Prior-year  end 

•  Natural  losses 

•  Enlisted 

•  End-of-year 

strength 

•  Force  shaping 

accessions 

strength 

losses 

•  Driven  by  NDAA/ 

fiscal  controls 

\ _ 

Adapted  from  Dave  Clark’s  2015  presentation  at  the  Naval  Postgraduate  School 
(personal  communication,  (September  10,  2015). 


Force  planners  at  N1  forecast  the  number  of  Sailors  who  will  leave  the 
Navy  each  year;  referred  to  as  manning  losses.  NRC’s  NCO  goal  ensures  that 
the  Navy’s  force  strength  meets  the  congressionally  mandated  end-strength  for 
each  FY  (Clark,  personal  communication,  2015).  Table  3  demonstrates  the 
different  phases  of  Navy  manning  within  a  FY.  The  NCO  mission  is  dynamic  and 
often  fluctuates  throughout  the  FY  in  response  to  actual  manning  losses. 


Table  3.  Navy  Manning  Terminology 


Terminology 

Description 

Begin  Strength 

Current  onboard  as  of  October  1  of  current  FY 

Force  Strength 

Current  onboard  anytime  between  October  2 
and  September  29  of  current  FY. 

End  Strength 

Current  onboard  as  of  September  30  of  current 
FY 

The  first  day  of  the  FY  is  October  1  and  the  last  day  of  the  FY  is  September  30. 


(2)  Loan  Repayment  Program 

The  loan  repayment  program  (LRP)  is  an  incentive  that  the  Navy  uses  to 
attract  high  quality  applicants  with  student  loan  debt  to  enlist  in  specific 
occupational  specialties.  Assuming  enlisted  service  members  with  less  than  one 
year  of  service  pursued  higher  education  prior  to  joining  the  military,  Figure  14 
indicates  that  approximately  60  percent  of  recruits  across  all  military  branches 
have  attended  at  least  some  college  before  joining  the  military.  Figure  14 
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includes  credits  towards  an  undergraduate  degree,  a  completed  undergraduate 
degree,  and  postgraduate  education. 


Figure  14.  Education  Levels  of  Enlisted  Personnel 


I  High-school  I  Some  college  O  4YR  degree* 


Assuming  that  service  members  who  have  between  0-1  year  of  service  enlisted 
into  the  service  with  the  education  level  shown.  Source:  Grefer,  Gregory,  and 
Rebhan  (2011,  p.  10). 

Through  the  LRP,  the  Navy  “pay[s]  federally  guaranteed  student  loans  (up 
to  $65,000)  through  three  annual  payments  during  a  Sailor’s  first  three  years  of 
service”  (Navy  Recruiting  Command,  2017).  Student  loan  debt  “is  the  only  form 
of  consumer  debt  that  has  grown  since  the  peak  of  consumer  debt  in  2008,”  as 
shown  in  Figure  15  (Lee,  2013,  p.  5). 
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Figure  15.  Rise  of  Student  Debt 


& 

&■ 


.& 


Source:  Lee  (2013,  p.  5). 


Consequently,  the  military’s  loan  repayment  program  may  be  an  attractive 
option  for  a  high  quality  recruit  with  student  loan  debt. 

(3)  High  School  Diploma 

Studies  indicate  that  a  high  school  diploma  is  “a  valuable  predictor  of 
military  attrition”  (Buddin,  1984,  p.  2).  Recruits  who  do  not  have  a  high  school 
diploma  are  more  likely  to  not  finish  their  initial  obligated  military  service.  In 
response  to  this  “well-known  result,”  the  DOD  has  a  benchmark  that  at  least  90 
percent  of  new  accessions  must  join  the  military  with  a  high  school  diploma 
(Navy  Recruiting  Command,  2007,  p.  7;  Buddin,  1984,  p.  1).  A  general  education 
development  (GED)  certificate  is  not  considered  a  high  school  diploma  (Buddin, 
1984,  p.  1). 

(4)  Recruit  Quality 

Recruit  quality  is  determined  by  an  applicant’s  Armed  Forces  Qualification 
Test  (AFQT)  score.  The  AFQT  score  is  derived  from  the  ASVAB’s  “Arithmetic 
Reasoning  (AR),  Mathematics  Knowledge  (MK),  Paragraph  Comprehension  (PC), 
and  Word  Knowledge  (WK)”  subsections  (Defense  Management  Data  Center, 
n.d.). 
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High  school  graduates  who  earn  above  a  50  AFQT  are  classified  as  “high 
quality”  applicants  (Navy  Recruiting  Command,  2007,  p.  6).  The  term  high  quality 
is  also  referred  to  as  Test  Score  Category  (TSC)  l-IIIA.  This  group  is  represented 
by  the  “A”  block  in  Figure  16. 

Figure  16.  Navy  Recruit  Quality  Determination 


AFQT 

TSC 

High  School 
Diploma 

No  High  School 
Diploma 

99-93 

1 

A 

B 

93-65 

II 

65-50 

IMA 

50-31 

IIIB 

Cu 

D 

31-0 

IVa 

Cl 

IVb 

IVc 

V 

DOD  Ineligible 

Adapted  from  Navy  Recruiting  Command  (2007,  p.  7). 

The  Navy  aims  to  recruit  applicants  who  meet  the  group  “A”  requirement 
because  they  qualify  for  most  Navy  occupational  specialties,  have  the  lowest  first 
term  attrition  rate,  historically  encounter  fewer  disciplinary  problems,  and  are  likely 
to  have  the  best  career  performance.  However,  this  category  of  applicants  tends  to 
be  the  most  expensive  to  recruit  (Navy  Recruiting  Command,  2007,  p.  7). 

High  quality  applicants  typically  have  multiple  opportunities,  such  as 
college  or  a  well-paying  job.  Therefore,  the  Navy  must  invest  more  money  in 
advertisements  targeting  group  “A,”  increase  enlistment  bonuses  to  incentivize 
group  “A,”  and  increase  recruiting  manpower  to  recruit  group  “A”  applicants. 
Each  of  these  contribute  to  the  high  cost  of  recruiting  high  quality  applicants. 

Descriptions  and  characteristics  of  all  categories  represented  in  Figure  16 
are  explained  in  Table  4. 
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Table  4.  Recruit  Quality  Category  Description 


Block/Category 

Description 

A 

(1)  Qualify  for  the  most  amount  of  programs 

(2)  Have  the  lowest  first  term  attrition 

(3)  Encounter  fewer  disciplinary  problems 

(4)  Likely  to  have  the  best  career  performance 

B 

(1)  Highest  first  term  attrition  rate 

(2)  Qualify  for  many  programs 

Cu 

(1)  Attrition  lower  than  “B,”  but  higher  than  “A” 

(2)  Applicants  do  not  gualify  for  many  programs. 

Cl 

Navy  does  not  recruit  from  this  group 

D 

Navy  does  not  recruit  from  this  group 

Adapted  from  Navy  Recruiting  Command  (2007,  p.  7). 


d.  PRO  Model:  Run  Options 

PRO  model  excursions  can  be  run  two  different  ways;  (1)  traditional  run, 
or  (2)  capacity  run. 

(1)  Traditional  Run 

The  traditional  run  option  of  the  PRO  model  performs  an  optimization  that 
minimizes  the  cost  of  recruiting  by  determining  the  optimal  allocation  of  resource 
spending  to  advertisements,  enlistment  bonuses,  education  incentives,  and 
recruiters.  The  traditional  run  can  be  evaluated  as  either  an  unconstrained  or  a 
constrained  problem. 

An  unconstrained  traditional  run  does  not  bound  any  of  the  decision 
variables.  The  result  is  “an  unconstrained,  minimum  cost  solution”  (Navy 
Recruiting  Command,  2007,  p.  22).  Unconstrained  traditional  runs  may  produce 
results  that  are  mathematically  feasible,  but  are  infeasible  in  practice.  For  example, 
while  it  would  be  unrealistic  for  NRC  to  have  more  than  4,000  recruiters  in  the  field, 
the  PRO  model  may  determine  4,520  recruiters  to  be  the  optimal  solution. 

Figure  17  shows  the  results  of  an  unconstrained  traditional  run  of  the  PRO 
model.  The  highlighted  rows  are  the  results.  In  FY  2015,  with  a  5  percent 
unemployment  rate,  and  a  recruiting  mission  of  34,000,  the  optimal  allocation  of 
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recruiting  resources  to  minimize  the  cost  of  recruiting  was:  assign  3,137  Sailors 
to  recruiting  duty,  allocate  $50,960,000  to  advertising,  and  $67,267,000  to 
enlistment  bonuses. 

Figure  17.  Results  of  an  Unconstrained  PRO  Model  Run 


Resource  Run 

2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCO 

34,000 

34,000 

34,000 

34.000 

34.000 

34.000 

34.325 

Capacity 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Unemployment  (%) 

5.0 

5.0 

5.0 

5.0 

5.0 

5.0 

5.0 

Float 

Total  Recruiters 

3,137 

3.132 

3.125 

3,117 

3.110 

3,101 

3,141 

Total  Recruiter  Cost  ($M) 

$256,944 

$269,364 

$262,626 

$265,932 

$269,281 

$273,186 

$281,505 

Float 

Advertising  ($M) 

$50,960 

$51,329 

$51,798 

$52,272 

$52  750 

$53,293 

$55,170 

Float 

Enlistment  Bonus  (SM) 

$67,267 

$67,754 

$68,374 

$68,999 

$69,630 

$70,346 

$72,825 

Float 

Education  Incentives  (SM) 

$0,000 

$0  000 

$0  000 

$0  000 

$0,000 

$0  000 

$0  000 

LRP  ($M) 

$7,440 

$11,220 

$11,280 

$11,380 

$11,430 

$11,460 

$11,670 

HSDG 

95% 

95% 

95% 

95% 

95% 

95% 

95% 

TSC  MIA 

70% 

70% 

70% 

70% 

70% 

70% 

70% 

Total  Cost  ($M) 

$382,611 

$389,667 

$394,078 

$398,583 

$403,091 

$408,285 

$421,170 

POM  FY17  version  of  the  PRO  model. 


In  contrast,  a  constrained  traditional  run  fixes  at  least  one  of  the  four 
decision  variables.  The  decision  variables  that  are  fixed  remain  constant.  The 
remaining  unconstrained  decision  variables  are  optimized  (Navy  Recruiting 
Command,  2007,  p.  22). 

Figure  18  demonstrates  the  results  of  a  constrained  traditional  PRO  model 
run  where  advertising  and  enlistment  bonus  were  fixed  and  the  total  number  of 
recruiters  was  optimized. 


Figure  18.  Results  of  a  Constrained  Traditional  PRO  Model  Run 


Resource  Run 

2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCO 

34,000 

34,000 

34,000 

34,000 

34,000 

34.000 

34,325 

Capacity 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Unemployment  (%) 

5.0 

5.0 

5.0 

5.0 

5.0 

5.0 

5.0 

Float 

Total  Recruiters 

3,037 

3,042 

3,047 

3,052 

3,058 

3,063 

3,151 

Total  Reciuiter  Cost  ($M) 

$248,724 

$251,880 

$256,097 

$260386 

$264,747 

$269,825 

$282408 

Fixed 

Advertising  ($M) 

$60,000 

$60000 

$60,000 

$60,000 

$60,000 

$60,000 

$60,000 

Fixed 

Enlistment  Bonus  ($14) 

$67,500 

$67,500 

$67,500 

$67,500 

$67,500 

$67,500 

$67,500 

Float 

Education  Incentives  ($14 ) 

$0,000 

$0,000 

$0,000 

$0,000 

$0,000 

$0,000 

$0,000 

LRP  (SM) 

$7,440 

$11,220 

$11,280 

$11,380 

$11,430 

$11,460 

$11,670 

HSDG 

95% 

95% 

95% 

95% 

95% 

95% 

95% 

TSC  MIA 

70% 

70% 

70% 

70% 

70% 

70% 

70% 

Total  Cost  (SM ) 

$383,664 

$390,600 

$394,877 

$399,266 

$403,677 

$408,785 

$421,578 

POM  FY17  version  of  the  PRO  model. 
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The  highlighted  rows  are  the  results.  In  FY  2015,  with  a  5  percent 
unemployment  rate,  a  recruiting  mission  of  34,000,  advertising  fixed  at 
$60,000,000,  and  enlistment  bonuses  fixed  at  $67,500,000,  the  Navy  should 
assign  3,037  Sailors  to  recruiting  duty. 

(2)  Capacity  Run 

The  capacity  run  estimates  the  number  of  recruits  the  Navy  can  expect  to 
recruit  based  on  a  predetermined  allocation  of  recruiting  resources.  Figure  19 
exhibits  the  results  of  a  capacity  run. 


Figure  19.  Results  of  a  Capacity  PRO  Model  Run 


Capacity  Run 

2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCO 

34.000 

34.000 

34.000 

34.000 

34.000 

34.000 

34.325 

Capacity 

33.426 

33,405 

33,385 

33.364 

33,344 

33,324 

33,303 

Unemployment  (%) 

5.0 

5.0 

5.0 

5.0 

5.0 

5.0 

5.0 

Total  Recruiters 

2.900 

2.900 

2.900 

2.900 

2,900 

2.900 

2,900 

Total  Recruiter  Cost  ($M) 

$237,520 

$240,119 

$243,725 

$247,381 

$251,093 

$255,477 

$259  917 

Advertising  ($M) 

$60,000 

$60  000 

$60,000 

$60  000 

$60,000 

$60,000 

$60,000 

Enlistment  Bonus  ($M) 

$67,500 

$67  500 

$67,500 

$67,500 

$67,500 

$67,500 

$67,500 

Education  Incentives  ($M) 

$0,000 

$0  000 

$0,000 

$0  000 

$0,000 

$0  000 

$0,000 

LRP  ($M) 

$7,440 

$7,440 

$11,220 

$11,280 

$11,380 

$11,430 

$11,460 

HSDG 

95% 

95% 

95% 

95% 

95% 

95% 

95% 

TSC  l-IIIA 

70% 

70% 

70% 

70% 

70% 

70% 

70% 

Total  Cost  ($M) 

$372,460 

$375,059 

$382,445 

$386,161 

$389  974 

$394,407 

$398  877 

POM  FY17  version  of  the  PRO  model. 


The  results  indicate  that  in  FY  2015  the  Navy  can  expect  33,426  recruits 
with  2,900  recruiters  in  the  field,  $60,000,000  allocated  to  advertising,  and 
$67,500,000  allocated  to  enlistment  bonuses. 

3.  Updates  to  the  PRO  Model 

In  201 1  the  SAG  Corporation  and  The  Lewin  Group,  Inc.  updated  the  PRO 
model  based  on  specific  shortcomings  of  the  model  identified  by  the  Navy  (Hogan 
et  al.,  n.d.,  p.  1).  The  updated  model  is  referred  to  as  the  Recruiting  Program 
Resource  Optimization  (E-PRO)  model.  The  E-PRO  model  added  “stochastic 
forecasting  capability”  and  updated  the  econometric  elasticities  within  the  recruiting 
cost  function  (Hogan  et  al.,  n.d.,  p.  3). 
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Despite  these  updates,  analysts  at  N1  still  use  the  PRO  model  since  it  is 
“simpler  in  construct  [compared  to  E-PRO]...  and  delivers  very  good  results” 
(Palmer,  personal  communication,  April  7,  2016).  As  mentioned  earlier,  the  current 
version  of  the  PRO  model  uses  “the  pooled  baseline  elasticities  updated  from 
the...  [2011]  E-Pro  effort”  (Palmer,  personal  communication,  April  12,  2016). 

4.  Limitations  of  the  PRO  Model 

The  existing  PRO  model  does  not  have  the  capability  to  efficiently  test 
uncertainties  in  variable  values,  or  the  effects  of  variable  interactions.  Without  this 
capability,  PRO  model  users  must  use  either  manual  trial  and  error  techniques  to 
test  different  scenarios  individually,  or  build  macros  in  Excel  to  test  the  fluctuation 
of  a  single  variable.  For  example,  a  macro  was  written  to  test  three  levels  of 
unemployment  rate,  as  shown  in  Table  5. 


Table  5.  Pooled  Unemployment  Rates 


FY  15 

FY1 6 

FY1 7 

FY1 8 

FY1 9 

FY  20 

FY21 

High  UE 

5.00% 

5.00% 

5.00% 

5.00% 

5.00% 

5.00% 

5.00% 

Base  UE 

4.50% 

4.50% 

4.50% 

4.50% 

4.50% 

4.50% 

4.50% 

Low  UE 

4.00% 

4.00% 

4.00% 

4.00% 

4.00% 

4.00% 

4.00% 

Adapted  from  POM  FY1 7  version  of  the  PRO  model. 


Without  options  for  multivariable  sensitivity  analysis  or  efficient 
experimentation,  it  is  difficult  to  understand  how  variable  interactions  or 
fluctuations  in  controllable  and  uncontrollable  factors  affect  the  model’s  output. 
This  may  be  an  area  of  concern  when  the  output  is  used  to  help  inform  decisions 
involving  hundreds  of  millions  of  dollars. 
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III.  METHODOLOGY  AND  IMPLEMENTATION 


This  chapter  covers  three  main  topics  that  span  the  motivation,  design, 
and  implementation  of  PROM-WED.  First,  an  overview  of  design  of  experiments 
techniques  is  presented.  Next,  the  field  of  data  farming  is  introduced  to  include 
examples  of  past  research  studies  that  have  utilized  data  farming.  Finally,  these 
two  concepts  are  integrated  as  the  design  and  construction  of  PROM-WED  is 
explained. 

A.  DESIGN  OF  EXPERIMENTS 

The  objective  of  an  experiment  across  any  discipline  of  study  is  “to 
investigate  characteristics  of  a  system”  (Park,  2007,  p.  309).  There  are  no  limits 
to  what  this  system  can  be,  from  the  test  and  evaluation  of  a  new  military 
warship,  to  sensitivity  analysis  on  a  political  science  poll.  Every  system  has 
inputs  and  outputs.  Inputs  are  either  controllable  or  uncontrollable.  Controllable 
factors  are  input  variables  to  the  system  that  are  known  and  can  be  set,  such  as 
the  number  of  Navy  destroyers  that  enter  a  theater  of  operations  in  a  combat 
simulation,  to  the  number  of  production  recruiters  Navy  Recruiting  Command  has 
in  the  continental  United  States.  Uncontrollable  factors  are  input  variables  to  the 
system  that  are  uncertain,  such  as  the  unemployment  rate  in  2021,  or  the 
probability  of  kill  for  an  adversary’s  new  weapon  system.  A  general  model  of  a 
system  is  shown  in  Figure  20. 


Figure  20.  General  Model  of  a  System. 


Inputs  ^ 


Controllable 


Outputs 


Uncontrollable 


Adapted  from  Penn  State  (n.d.). 
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Early  development  in  DOE  methodology  occurred  predominately  in  the 
physical  sciences,  specifically  in  agriculture  (Penn  State,  n.d.).  The  classical 
methods  and  foundations  of  DOE  can  also  be  applied  to  the  testing  and  analysis 
of  simulation  models  (Sanchez,  2006,  p.  69).  Control,  replication,  and 
randomization  are  considered  to  be  “fundamental  concepts”  of  DOE  (Sanchez, 
2006,  p.  69).  Working  definitions  of  these  concepts  in  the  context  of  DOE  are 
shown  in  Table  6. 


Table  6.  Fundamental  Concepts  of  DOE. 


Fundamental 

Concept 

Working  Definition 

Control 

“The  experiment  is  conducted  in  a  systematic  manner  after 
explicitly  considering  potential  sources  of  error,  rather  than  by 
using  a  trial-and-error*  approach.  ” 

Replication 

“A  way  to  gain  enough  data  to  achieve  narrow  confidence 
intervals  and  powerful  hypothesis  tests.” 

Randomization 

“Provides  a  probabilistic  guard  against  the  possibility  of  unknown, 
hidden  sources  of  bias  surfacing  to  create  problems  with  your 
data.” 

Adapted  from  Sanchez  (2006,  69). 


To  adequately  test  a  system,  whether  the  system  is  a  simulation  model  or 
a  physical  science  experiment,  trial-and-error  should  be  avoided.  Trial-and-error 
is  inefficient,  difficult  to  replicate,  and  lacks  control.  DOE  techniques  combat 
these  limitations  through  systematically  testing  a  model  with  control,  replication, 
and  randomization.  Systematic  approaches  are  also  conducive  for  automation, 
which  alleviates  manual  work,  and  increases  the  efficiency  and  capability  of  the 
system  being  explored.  The  automation  of  DOE  techniques  has  created  the  field 
of  data  farming,  which  is  further  explained  in  the  next  section. 

There  are  many  different  DOE  methods  and  techniques  available,  such  as 
the  full  and  fractional  factorials,  central  composite  designs,  and  nearly  orthogonal 
Latin  hypercubes  (NOLHs).  The  full  factorial  and  NOLH  methods  are  explained  in 
further  detail.  More  information  regarding  DOE  basic  concepts,  methods  and 
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their  application  to  simulation  modeling  can  be  found  in  Sanchez  and  Wan’s 
report,  “Work  Smarter,  Not  Harder:  A  tutorial  on  designing  and  conducting 
simulation  experiments”  (Sanchez  &  Wan,  2015). 

1.  Full  Factorial  DOE  Method 

The  full  factorial  approach  tests  every  possible  combination  of  input 
factors  given  fixed  levels.  The  classic  game  of  “capture  the  flag”  is  used  to 
explain  the  full  factorial  method.  The  objective  of  the  game  is  for  a  member  of 
one  team  to  capture  a  flag  that  is  kept  on  the  other  side  of  the  field,  and  return  it 
to  their  side  of  the  field.  If  caught  by  a  member  of  the  opposing  team  on  the 
opposition’s  side  of  the  field,  the  player  fails  the  mission,  and  is  temporarily 
placed  in  “jail.”  Figure  21  shows  a  simple  representation  of  the  “capture  the  flag” 
game,  where  the  gray  team  on  the  left  is  trying  to  capture  the  gray  flag  on  the 
opposition’s  side,  and  vice  versa. 


Figure  21 .  Capture  the  Flag  Game 


The  circles  represent  the  players  of  each  team.  Adapted  from  MultiCulturalGames 
(nd.). 

Two  attributes  that  may  affect  the  success  of  a  “capture  the  flag”  player 
are  speed  and  stealth.  Figure  22  illustrates  the  testing  of  various  degrees  of 
speed  and  stealth  for  a  “capture  the  flag”  player.  The  sparse  grid  on  the  left  tests 
the  system  only  at  its  extreme  values,  where  either  minimum  speed  or  minimum 
stealth  results  in  a  failure,  but  maximum  speed  and  maximum  stealth  results  in  a 
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success.  The  grid  on  the  right  demonstrates  a  dense  full  factorial  test  where 
many  possible  levels  of  stealth  are  tested  against  many  possible  levels  of  speed. 
In  this  hypothetical  example,  success  can  be  met  at  something  other  than  a 
combination  of  full  speed  and  full  stealth  (Sanchez  &  Wan,  2015,  p.  1801). 


Figure  22.  A  Sparse  versus  a  Dense  Full  Factorial  DOE  for 

Capture  the  Flag 


Speed 


►GOO 
▼▼▼▼OOOOO 
TTTTOOOOO 

v v vvooooo 
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The  block  shapes  indicate  failure,  whereas  the  circle  indicates  success.  The 
triangle  represents  a  result  somewhere  in  between  failure  and  success.  Source: 
Sanchez  and  Wan  (2015,  p.  1801). 


The  dense  full  factorial  grid  in  Figure  22  illustrates  two  key  advantages 
that  DOE  techniques  can  offer:  (1)  space-filling  capability,  and  (2)  robust  insight 
and  understanding  of  the  solution  space. 

Space-filling  refers  to  a  DOE’s  capability  of  testing  the  simulation  over  a 
broad  spectrum  of  input  combinations  (Sanchez  &  Wan,  2015).  Figure  22 
demonstrates  that  testing  only  the  maximum  and  minimum  values  does  not  have 
good  space-filling  capability,  whereas  using  the  multi-level  full  factorial  DOE 
exemplifies  high  space-filling  capability.  The  ability  to  test  a  factor  at  different 
levels  increases  the  potential  insight  gained  from  the  solution  space  (Sanchez  & 
Wan,  2015).  As  demonstrated  in  Figure  22,  the  space-filling  DOE  provides 
insight  to  capture  the  flag  players  that  the  right  combination  of  stealth  and  speed 
resources  can  achieve  the  target  solution  using  less  resources.  Full  factorial 
DOEs  are  orthogonal,  which  means  that  there  are  no  confounding  effects. 
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Certain  combinations  of  input  variables,  such  as  speed  and  stealth,  may 
influence  the  effect  of  each  other.  This  is  referred  to  as  a  variable  interaction. 
Variable  interactions  identify  “whether  the  levels  of  some  factors  influence  the 
effects  that  other  factors  have”  on  the  solution  (Sanchez  &  Wan,  2015,  p.  1796). 
Without  proper  care  in  designing  experiments,  interactions  can  be  impossible  to 
estimate. 

Time  and  computing  capability  can  quickly  become  limiting  factors  when 
performing  DOE  tests  on  complex  simulation  models.  Testing  a  complex  model 
using  a  full  factorial  of  all  possible  combinations  of  variables  is  inefficient  and 
often  inconceivable.  For  example,  Table  7  demonstrates  how  a  DOE  that 
examines  a  model  with  only  20  factors  can  quickly  become  infeasible  as  the 
number  of  levels  increases. 


Table  7.  Number  of  Experiments  Required  to  Test  a  Model  with  20 

Factors  Using  Full  Factorial  Designs 


Number  of  levels  each  factor  is 
studied  at 

Equation 

Number  of  Experiments  Required 

2 

(i.e.,  only  a  min  and  max  value) 

o 

CM 

CM 

1,048,576 

4 

(i.e.,  min,  max  and  2  values  in 
between) 

o 

CM 

1,099,510,000,000 

6 

(i.e.,  min,  max  and  4  values  in 
between) 

O 

CM 

CD 

365,616,000,000,000 

Adapted  from  Sanchez  (2006,  p.  76). 

Increasing  the  number  of  experiments  becomes  costly  since  more 
experimental  runs  require  higher  computing  capability,  and  increased  work 
hours. 
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2. 


Nearly  Orthogonal  Latin  Hypercubes 


Cioppa  and  Lucas  (2007)  developed  the  nearly  orthogonal  Latin 
hypercubes  (NOLH)  which  are  efficient  and  effective  alternatives  to  the  full 
factorial  DOE.  “Latin  hypercube  designs  have  proven  useful  for  exploring 
complex,  high-dimensional  computational  models,  but  can  be  plagued  with 
unacceptable  correlations  among  input  variables”  (Hernandez,  Lucas,  &  Carlyle, 
2012,  p.  1).  Cioppa  and  Lucas’  work  addresses  this  problem  by  “inducing  small 
correlations  between  the  columns  in  the  design  matrix”  (2007,  p.  45).  The  result 
is  the  nearly  orthogonal  Latin  hypercube.  These  NOLH  DOEs  provide  analysts 
with  many  advantages,  including  the  ability: 

to  determine  the  driving  factors,  detect  interactions  between  input 
variables,  identify  points  of  diminishing  or  increasing  rates  of  return, 
and  find  thresholds  or  change  points  in  localized  areas...  [and]  fit 
many  diverse  metamodels  to  multiple  outputs  with  a  single  set  of 
runs.  (MacCalman,  Vieira,  &  Lucas,  2016,  p.  1) 

Figure  23  shows  a  comparison  of  space-filling  capabilities  between  two  full 
factorial  designs  (A  and  B),  versus  two  NOLH  designs  (C  and  D).  The  four 
designs  are  respectively  a  24  and  a  44  full  factorial  designs,  and  a  17-point  and 
257-point  NOLH  DOEs. 


Figure  23. 
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Source:  Sanchez  and  Wan  (2015,  p.  1802). 


Table  8  provides  a  numerical  representation  of  the  four  DOE  designs 
shown  in  Figure  23.  For  one  extra  design  point  (i.e.,  16  to  17,  or  256  to  257),  we 

get  much  greater  space  filling  with  the  NOLH  DOE. 
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Table  8.  Factorial  Designs  versus  NOLH  Designs 


Pairwise  Plot 
Matrix 

Design 

Factors 

Levels 

Design 

Points 

A 

24  Factorial 

4 

2 

16 

B 

44  Factorial 

4 

4 

256 

C 

NOLH 

7 

17 

17 

D 

NOLH 

29 

257 

257 

Adapted  from  Sanchez  and  Wan  (2015,  p.  1802). 


As  demonstrated  by  Figure  23  and  Table  8,  the  NOLFi  designs  minimize 
computational  effort  while  improving  space-filling  capability,  allowing  for  more 
factors  to  be  tested  within  the  same  experimental  design  (Sanchez  &  Wan,  2015, 
p.  1803).  At  the  cost  of  one  additional  design  point,  we  are  able  to  analyze  7  or 
29  factors  at  17  and  257  levels,  respectively,  in  comparison  to  a  factorial  design 
with  4  factors  at  either  2  or  4  levels.  Reference  Cioppa  and  Lucas’  paper 
“Efficient  Nearly  Orthogonal  and  Space-filling  Latin  Hypercubes”  for  more 
information  about  the  NOLH  DOE  method  (Cioppa  &  Lucas,  2007). 

From  the  initial  research  done  by  Cioppa  and  Lucas,  other  families  of 
NOLH  designs  have  been  developed  to  enhance  and  make  the  NOLH  designs 
adaptable  to  further  applications  in  simulation  analysis.  To  expand  the  NOLH 
designs  capability  a  mixed  integer  program  (MIP)  algorithm  was  developed  “that 
generates  Latin  hypercubes  with  little  or  no  correlation  among  their  columns  for 
most  any  determinate  run-variable  combination”  (Hernandez  et  al.,  2012,  p.  1). 
This  MIP  algorithm  is  also  adaptable  and  accommodating  to  run  modifications. 
(Hernandez  et  al.,  2012,  p.  1).  A  second-order  NOLH  design  has  also  been 
developed  that  facilitates  “exploratory  analysis  of  stochastic  simulation  models  in 
which  there  is  considerable  a  priori  uncertainty  about  the  forms  of  the  responses” 
(MacCalman  et  al.,  2016,  p.  1).  Lastly,  Sanchez  created  a  Microsoft  Excel 
spreadsheet  that  uses  Cioppa  and  Lucas’  NOLH  DOE  algorithm  to  provide  users 
with  the  ability  to  generate  designs  ranging  from  simple  small  orthogonal  Latin 
hypercubes  to  complex  NOLH  designs  that  handle  up  to  29  factors  at  257  levels 
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each  (Sanchez,  2011).  These  designs,  along  with  other  DOE  methods,  are 
available  in  Microsoft  Excel  format  at  https://harvest.nps.edu. 

B.  DATA  FARMING 


Work  smarter,  not  harder... 

— Professor  Susan  Sanchez  (2006) 


The  use  of  robust  design  of  experiment  techniques  has  spawned  a  field  of 
data  analytics  for  simulation  models,  referred  to  as  data  farming.  In  comparison 
to  traditional  methods  such  as  data  mining,  where  one  “seek[s]  to  uncover 
valuable  nuggets  of  information  buried  within  massive  amounts  of  data,”  data 
farming  grows  data  by  controlling  the  interactions  of  the  variables  through 
efficient  DOE  techniques  (Sanchez,  2014,  p.  800).  Retrospective  data  collection 
can  find  correlations,  but  prospective  DOE  is  required  to  establish  causality. 

Data  farming  is  an  iterative  process  that  allows  analysts  to  gain  robust 
insight  into  the  ‘“big  picture’  solution  landscape”  (Horne  &  Meyer,  2010,  p.  1).  Six 
foundational  components  of  data  farming  are  shown  in  Figure  24. 


Figure  24.  The  Six  Realms  of  Data  Farming 
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Source:  Horne  and  Meyer  (2010,  p.  2). 
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Steve  Upton  of  the  SEED  center  at  NPS  has  built  multiple  data  farming 
wrappers  to  facilitate  efficient  DOE  testing  around  simulation  models  spanning 
diverse  computing  environments  and  subject  areas.  The  data  farming  wrappers 
that  he  builds  are  computer  programs  that  wrap  a  DOE  algorithm  around  a  pre¬ 
existing  model.  The  following  is  a  sample  of  research  that  utilizes  Upton’s  data 
farming  wrappers: 

1 .  Erin  Borozny  tested  the  Navy’s  Officer  Strategic  Analysis  Model 
(OSAM)  using  data  farming.  OSAM  is  a  manpower  model  that 
projects  officer  end  strength  and  force  structure  based  on 
“personnel  plans  and  force-shaping  policy”  (Borozny,  2015,  p.  v). 
Her  research  provides  insight  into  effective  ways  the  Navy  can 
better  manage  its  officer  inventory  in  order  to  meet  authorized  end 
strength  at  the  end  of  each  FY  (Borozony,  2015). 

2.  Christian  Seymour  applied  data  farming  to  the  Synthetic  Theater 
Operations  Research  Model  (STORM).  The  Department  of  Defense 
uses  STORM  as  its  “primary  campaign  analysis  tool”  that  considers 
“force  structures,  operational  concepts,  and  military  capabilities” 
(Seymour,  2014,  p.  v).  His  study  shows  that  data  farming 
“capitalize^]  on  STORM’S  full  potential”  and  provides  policy  makers 
with  robust  insights  in  an  efficient  and  effective  manner  (Seymour, 

2014,  p.  v). 

3.  Jeffery  Parker’s  research  on  the  Marine  Corps’  future  amphibious 
capability  used  data  farming  around  a  model  that  simulated 
amphibious  assaults.  His  research  provides  informative  decision 
support  for  United  States  Navy  procurement  “by  evaluating  the 
[Marine  Expeditionary  Unit’s]  MEU’s  expeditionary  amphibious 
assault  capability  and  the  use  of  ship-to-shore  connectors”  (Parker, 

2015,  p.  v). 

These  are  only  three  examples  of  numerous  studies  that  have  utilized  a 
data  farming  wrapper  around  a  simulation  model.  They  demonstrate  how 
adaptable,  capable,  and  valuable  data  farming  an  existing  model  can  be.  For 
more  information  about  studies  that  have  used  data  farming  in  defense 
applications,  visit  https://harvest.nps.edu. 
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c. 


PROM-WED 


PROM-WED  was  developed  to  provide  analysts  with  a  tool  that  evaluates 
the  PRO  model  over  scenarios  constructed  using  the  NOLH  DOE  algorithm. 
PROM-WED  also  provides  analysts  with  decision  support  capabilities  that 
capitalize  on  its  ability  to  grow  data,  and  perform  sensitivity  and  risk  analysis  to 
better  inform  decision  makers  on  a  robust  solution  to  the  optimal  allocation  of 
recruiting  resources.  PROM-WED  excursions  can  be  run  to  model  the  effects  of 
varying  degrees  of  policy  changes  and  a  range  of  economic  and  demographic 
conditions  that  affect  the  total  cost  of  recruiting.  One  PROM-WED  excursion 
provides  decision  support  analysis  to  cover  the  effects  of  all  of  these  factors  and 
their  interactions  with  one  another. 

To  achieve  these  objectives,  PROM-WED  is  divided  into  three  main 
components:  (1)  the  NOLH  DOE  data  farming  wrapper,  (2)  the  GUI,  and  (3) 
decision  support  analysis.  For  the  purpose  of  this  research,  focus  is  placed  on 
the  traditional  run  option.  Refer  to  Chapter  V  regarding  the  capacity  run  option. 

Since  the  PRO  model  is  built  in  Microsoft  Excel,  PROM-WED  is  also  built 
in  Microsoft  Excel,  specifically  Microsoft  Excel  2013  Version  15.0.4849.1003 
(Microsoft  Excel,  2013).  Given  the  restrictions  and  limitations  of  software  allowed 
on  government  computers,  maintaining  PROM-WED  in  the  Microsoft  Excel 
environment  allows  accessibility  of  use  to  any  government  computer  without 
requiring  any  additional  software. 

1.  Data  Farming  Wrapper 

The  NOLH  DOE  algorithm  is  the  foundation  of  PROM-WED’s  data  farming 
wrapper.  The  NOLH  was  chosen  for  its  space-filling  capability  and  ease  of  use  in 
a  Microsoft  Excel  VBA  modeling  environment.  The  SEED  Center  at  NPS  has 
made  the  NOLH  DOE  algorithm  available  in  a  Microsoft  Excel  worksheet  at 
https://harvest.nps.edu. 

PROM-WED’s  data  farming  wrapper  uses  both  the  33-point  and  129-point 

NOLH  design  worksheets.  The  33-point  design  tests  up  to  11  variables  at  33 
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levels,  whereas  the  129-point  design  tests  up  to  22  variables  over  129  levels. 
The  129-point  design  has  better  space-filling  properties,  but  takes  more  time  to 
run.  Figure  25  shows  a  pairwise  plot  comparison  of  the  space-filling  ability  of 
these  two  designs.  The  user  is  able  to  choose  which  NOLH  design  they  want  to 
run  excursions  over  using  the  GUI  that  is  further  explained  in  the  next  section. 


Figure  25.  Pairwise  Plots  for  the  33  and  1 29  Point  NOLH  Designs 


Left:  33-point  NOLH  DOE.  Right:  129-point  NOLH  DOE. 


Table  9  shows  an  example  PROM-WED  test  case  scenario. 


Table  9.  Example  PROM-WED  Scenario 


Variable  Type 

Variable  Name 

Value  Low 

Value  High 

Decision  Variable 

Production  Recruiters 

2,500  recruiters 

3,500  recruiters 

Market  Factor 

Unemployment  Rate  (UE) 

4.0% 

8.0% 

Market  Factor 

Relative  Pay 

0.8 

1.2 

Policy  Factor 

Recruiting  Mission  (NCO) 

30,000  recruits 

40,000  recruits 

Figure  26  shows  the  implementation  of  this  scenario  in  the  33-point  NOLH 
design  worksheet.  A  129-point  NOLH  design  worksheet  can  be  found  in 
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Appendix  A.  Each  FY  that  is  explored  has  its  own  worksheet  similar  to  the  one 
seen  in  Figure  26  for  FY  2017.  PROM-WED  provides  users  with  a  recruiting 
resource  allocation  over  seven  FYs.  Therefore,  there  are  seven  33-point  NOLH 
design  worksheets  and  seven  129-point  NOLH  design  worksheets  within  PROM- 
WED’s  data  farming  wrapper. 


Figure  26.  Scenario  Inputted  into  the  NOLH  Worksheet 


A 

B 

c 

D 

E 

F 

G 

H 

J 

K 

L 

1 

low  level 

0.0001 

0.04 

2500 

11.28 

34.8264 

40.97 

30000 

0.7  r 

0.95 

0.8 

1883304 

2 

high  level 

0.0001 

0.08 

3500 

11.28 

34.8264 

40.97 

40000 

0.7 " 

0.95 

1.2 

1883304 

3 

decimals 

4 

3 

0 

3 

3 

3 

0 

2 

2 

6 

0 

4 

factor  name 

NCF  +  College  First  Unemployment  Rate  Recruiters 

LRP 

Advertising  (AC  Enl.  Only) 

EB 

NCO  (50%  BoY  DEP)  TSC  l-IIIA 

HSDG  ative  Pay 

QMA 

5 

0.0001 

0.044 

2938 

11.28 

34.826 

40.97 

36875 

0.7 

0.95 

1.075 

1883304 

6 

FY 

0.0001 

0.08 

2625 

11.28 

34.826 

40.97 

37500 

0.7 

0.95 

0.975 

1883304 

7 

2017 

0.0001 

0.058 

3406 

11.28 

34.826 

40.97 

37188 

0.7 

0.95 

1  1875 

1883304 

8 

0.0001 

0.075 

3500 

11.28 

34.826 

40.97 

38125 

0.7 

0.95 

0.8375 

1883304 

9 

0.0001 

0.041 

2969 

11.28 

34.826 

40.97 

34063 

0.7 

0.95 

0.85 

1883304 

10 

0.0001 

0.078 

2813 

11.28 

34.826 

40.97 

31563 

0.7 

0.95 

1.0125 

1883304 

11 

0.0001 

0.059 

3469 

11.28 

34.826 

40.97 

33750 

0.7 

0.95 

0.8 

1883304 

12 

0.0001 

0.068 

3438 

11.28 

34.826 

40.97 

32188 

0.7 

0.95 

1.175 

1883304 

13 

0.0001 

0.05 

2719 

11.28 

34.826 

40.97 

30000 

0.7 

0.95 

1.0375 

1883304 

14 

0.0001 

0.066 

2781 

11.28 

34.826 

40.97 

30938 

0.7 

0.95 

0.875 

1883304 

15 

0.0001 

0.049 

3250 

11.28 

34.826 

40.97 

31250 

0.7 

0.95 

1.05 

1883304 

16 

0.0001 

0.069 

3156 

11.28 

34.826 

40.97 

34688 

0.7 

0.95 

0.8625 

1883304 

17 

0.0001 

0.046 

2688 

11.28 

34.826 

40.97 

39688 

0.7 

0.95 

0.9125 

1883304 

18 

0.0001 

0.064 

2875 

11.28 

34.826 

40.97 

39375 

0.7 

0.95 

1.1125 

1883304 

19 

0.0001 

0.048 

3344 

11.28 

34  826 

40.97 

36563 

0.7 

0.95 

0.9375 

1883304 

20 

0.0001 

0.065 

3094 

11.28 

34.826 

40.97 

35625 

0.7 

0.95 

1.1 

1883304 

21 

0.0001 

0.06 

3000 

11.28 

34.826 

40.97 

35000 

0.7 

0.95 

1 

1883304 

22 

0.0001 

0.076 

3063 

11.28 

34.826 

40  97 

33125 

0.7 

0  95 

0.925 

1883304 

23 

0.0001 

0.04 

3375 

11.28 

34.826 

40.97 

32500 

0.7 

0.95 

1.025 

1883304 

24 

0.0001 

0.063 

2594 

11.28 

34.826 

40.97 

32813 

0.7 

0.95 

0.8125 

1883304 

25 

0.0001 

0.045 

2500 

11.28 

34.826 

40.97 

31875 

0.7 

0.95 

1.1625 

1883304 

26 

0.0001 

0  079 

3031 

11.28 

34.826 

40.97 

35938 

0.7 

0.95 

1.15 

1883304 

27 

0.0001 

0.043 

3188 

11.28 

34.826 

40  97 

38438 

0.7 

0.95 

0.9875 

1883304 

28 

0.0001 

0.061 

2531 

11.28 

34.826 

40.97 

36250 

0.7 

0.95 

1.2 

1883304 

29 

0.0001 

0.053 

2563 

11.28 

34.826 

40.97 

37813 

0.7 

0.95 

0.825 

1883304 

30 

0.0001 

0.07 

3281 

11.28 

34  826 

40.97 

40000 

0.7 

0.95 

0.9625 

1883304 

31 

0.0001 

0.054 

3219 

11.28 

34.826 

40.97 

39063 

0.7 

0.95 

1.125 

1883304 

32 

0.0001 

0.071 

2750 

11.28 

34.826 

40.97 

38750 

0.7 

0.95 

0.95 

1883304 

33 

0.0001 

0.051 

2844 

11.28 

34.826 

40  97 

35313 

0.7 

0.95 

1.1375 

1883304 

34 

0.0001 

0.074 

3313 

11.28 

34.826 

40.97 

30313 

0.7 

0.95 

1.0875 

1883304 

35 

0.0001 

0.056 

3125 

11.28 

34  826 

40.97 

30625 

0.7 

0.95 

0.8875 

1883304 

36 

0.0001 

0.073 

2656 

11.28 

34.826 

40.97 

33438 

0.7 

0.95 

1.0625 

1883304 

37 

0.0001 

0  055 

2906 

11.28 

34.826 

40.97 

34375 

0.7 

0.95 

0.9 

1883304 

Figure  26  illustrates  that  each  input,  whether  it  be  a  controllable  or 
uncontrollable  variable,  is  tested  over  33  levels.  Recruiting  mission,  number  of 
recruiters,  UE,  and  relative  pay  are  the  variables  that  are  tested  over  a  range  of 
values.  The  lower  bound  on  the  range  is  fed  into  the  “low  level”  cell,  whereas  the 
upper  bound  on  the  range  is  fed  into  the  “high  level”  cell.  For  the  variables  that 
remain  constant,  the  low  and  high  values  are  the  same.  The  “decimals”  cell 
refers  to  the  number  of  significant  digits  in  the  decimal  place  that  the  NOLH 
algorithm  divides  the  factor  into.  For  example,  recruiters,  NCO,  and  QMA 
variables  all  have  a  zero  in  the  “decimals”  cell  since  these  variables  represent 
people,  and  having  a  fraction  of  a  person  is  infeasible. 
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Each  row  of  the  worksheet  shown  in  Figure  26  represents  a  different 
scenario.  A  subroutine  loops  over  each  row  of  the  worksheet  and  feeds  the 
values  for  each  input  variable  into  the  legacy  PRO  model.  The  subroutine  that 
executes  33  design  point  NOLH  excursions  can  be  found  in  Appendix  B.  The 
legacy  PRO  model’s  “RunTraditional”  macro  was  adapted  to  accommodate  data 
farming.  The  modified  macro  is  now  referred  to  as  “RunTraditional6.” 

A  33-design  point  NOLH  design  will  result  in  33  different  legacy  PRO 
model  solutions,  and  a  129-design  point  NOLH  design  will  result  in  129  different 
legacy  PRO  model  solutions.  The  NOLH  worksheet  married  with  the  subroutine 
makes  up  the  data  farming  wrapper. 

2.  Graphical  User  Interface 

PROM-WED’s  GUI  makes  data  farming  easily  accessible  to  any  PRO 
model  user  regardless  of  knowledge  or  skill  in  DOE  techniques  or  data  farming. 
A  snapshot  of  PROMWED’s  GUI  is  shown  in  Figure  27. 


Figure  27.  PROM-WED’s  GUI 
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The  variables  are  categorized  as  either  “Decision  Variables”  or  “Market 
Factors.”  A  decision  variable  can  either  be  constrained  (“Fixed”)  or  unconstrained 
(“Floated”).  The  title  “Market  Factors”  is  a  blanket  category  that  covers  both 
market  factors,  as  well  as  policy  factors,  as  described  in  Chapter  II. 

A  brief  description  of  how  a  PROM-WED  excursion  is  performed  using  the 
GUI  is  now  presented.  A  detailed  PROM-WED  user  manual  can  be  found  in 
Appendix  C. 

To  constrain  a  decision  variable,  select  the  variable  of  interest  and  click  on 
“Fix  DV.”  A  constrained  decision  variable  can  either  be  fixed  as  a  constant  or 
tested  over  a  range  of  values  using  the  NOLFI  algorithm.  If  the  user  is  interested 
in  testing  over  a  range,  the  desired  lower  and  upper  bounds  of  the  range  are 
inputted  into  the  “Design  of  Experiments  Table,”  as  shown  in  Figure  28. 


Figure  28.  Testing  a  Decision  Variable  over  a  Constrained  Range 
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Each  “low  level”  and  “high  level”  value  of  the  “Design  of  Experiments 
Table”  is  linked  to  a  NOLH  worksheet.  For  example,  the  low  and  high  values  for 
FY  17  are  linked  to  the  NOLH  worksheet  for  FY  2017,  as  shown  previously  in 
Figure  26. 

A  similar  procedure  is  followed  for  each  variable  listed  in  the  “Market 
Factors”  category.  The  user  must  work  through  each  variable  in  the  “Market 
Factors”  list,  and  choose  whether  it  is  kept  constant  (“Fix  Value”),  or  tested  over 
a  range  of  values  (“Set  Range”).  The  NOLH  DOE  is  complete  once  all  variables 
listed  in  the  “Market  Factors”  category  are  accounted  for.  Once  the  NOLH  is  fully 
populated,  as  shown  in  Figure  29,  the  user  selects  the  “Run  Type,”  and  the 
number  of  design  points  the  NOLH  is  tested  over.  Currently  PROM-WED  has  the 
capability  to  test  the  traditional  run  option.  Further  work  is  required  for  the 
capacity  run  option. 


Figure  29.  PROM-WED  GUI  when  NOLH  is  Fully  Populated 
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Selecting  the  “NOLH  RUN”  button  executes  the  subroutine  to  begin 
growing  the  data.  The  33-point  design  takes  approximately  two  minutes  to  run  on 
a  standard  modern  personal  computer  (PC),  whereas  the  129-point  design  takes 
about  five  to  ten  minutes  to  run.  Run  times  are  dependent  on  factors  such  as  the 
operating  system  and  computational  capacity  of  the  computer.  The  result  for 
each  PROM-WED  scenario  is  deposited  to  a  worksheet  for  further  analysis. 

3.  PROM-WED  Decision  Support  Analysis 

In  addition  to  growing  PRO  model  data  using  data  farming,  PROM-WED 
provides  users  with  decision  support  capabilities  to  analyze  the  data  grown  by 
each  excursion.  PROM-WED  offers  two  decision  support  capabilities:  (1) 
automatically  generated  analysis  and  (2)  data  generated  for  further  analysis 
requiring  a  statistical  software  package.  In  this  section,  PROM-WED’s  decision 
support  capabilities  are  discussed.  The  focus  is  on  why  each  type  of  graph  or 
table  was  chosen.  Chapter  IV  has  a  detailed  discussion  dedicated  to  analyzing 
PROM-WED’s  decision  support  capability. 

a.  Automatically  Generated  Decision  Support  Capability 

The  purpose  of  PROM-WED’s  automatically  generated  decision  support 
analysis  is  to  provide  users  with  a  tool  capable  of  providing  an  at-a-glance 
understanding  of  the  solution  space  of  a  completed  PROM-WED  excursion. 
PROM-WED’s  “Decision  Support  Analysis”  for  the  traditional  run  option  provides 
users  with  a  broad  understanding  of  how  variability  in  decision  variables, 
controllable  policy  changes,  and  uncontrollable  market  factors  affect  the  total 
cost  of  recruiting.  Since  the  traditional  run  addresses  the  allocation  of  resources 
(i.e.,  the  decision  variables),  the  automatically  generated  decision  support 
capability  provides  at-a-glance  insights  to  decision  makers  regarding  the  optimal 
allocation  of  recruiting  resources  using  the  33-point  design.  In  the  next  section, 
further  insights  regarding  variable  interactions  and  the  effects  of  the  various 
market  factors  are  explored  using  a  commercial  statistical  software  package. 
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In  an  effort  to  provide  as  much  relevant  information  as  possible  within  an 
easily  printed  worksheet,  Figures  30  and  31  show  the  two  pages  that  comprise 
PROM-WED’s  automatically  generated  decision  support  capability  for  the 
traditional  run  option. 


Figure  30.  Traditional  Run  Decision  Support  Analysis,  Page  1 
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Figure  31 .  Traditional  Run  Decision  Support  Analysis,  Page  2 
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The  six  graphs  and  one  table  make  up  the  traditional  run’s  decision 
support  analysis.  The  purpose  of  each  graph  is  now  explained. 
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Starting  in  the  top  left,  the  “Mean  Total  Cost  of  Recruiting  over  FYDP” 
graph,  also  shown  in  Figure  32,  shows  the  resulting  mean  total  cost  of  recruiting 
for  each  FY. 

Figure  32.  Graph  1 :  Mean  Total  Cost  of  Recruiting  over  FYDP 
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The  mean  for  each  FY  is  represented  by  the  blue  dots.  The  red  dashed 
lines  represent  the  95  percent  confidence  interval  for  each  mean.  Where  “n”  is 
the  number  of  sample  points.  For  example,  n  =  33  for  the  33-point  NOLH  design, 
and  so  forth.  Here  we  are  treating  each  observation  as  an  equally  likely  sample 
of  possible  recruiting  scenarios.  The  95  percent  confidence  intervals  for  all 
graphs  shown  in  the  automatically  generated  decision  support  analysis  are 
calculated  as  follows: 


(1 )  First,  the  sample  standard  deviation  is  calculated: 


s  = 


The  Microsoft  Excel  formula  STDEV.SQ  is  used  in  PROM-WED. 
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(2)  Next,  since  each  scenario  is  independent,  and  it  is  assumed  that 
the  sample  mean  is  approximately  normally  distributed,  the  margin 
of  error  at  0.05  significance  level  is  calculated: 


s 

margin  of  error  =  za/2  -p; 

Vn 

where :  a  =  0.05  and 

za/2  is  the  100(1  —  a/2)  percentile  of  a 
standard  normal  random  variable 


The  Microsoft  Excel  formula  CONFIDENCE. NORMQ  is  used  in  PROM-WED. 


(3)  Finally,  the  upper  and  lower  confidence  bounds  are  calculated: 

x  ±  ynargin  of  error 

The  region  between  the  two  red  dashed  lines  represents  with  95  percent 
confidence  the  mean  total  cost  of  recruiting  is  somewhere  within  this  range. 

The  second  graph  “PROM-WED  Excursion  versus  Program  of  Record,” 
also  shown  in  Figure  33,  compares  the  mean  optimal  allocation  of  recruiting 
resources  that  resulted  from  the  PROM-WED  excursion  with  the  program  of 
record. 


Figure  33.  Graph  2:  PROM-WED  Excursion  versus  Program  of  Record 
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The  program  of  record  (POR)  is  the  resource  allocation  “recorded  in  the 
current  Future  Years  Defense  Program  (FYDP)  or  as  updated  from  the  last 
FYDP  by  approved  program  documentation”  (DAU,  n.d.).  Within  the  legacy  PRO 
model  the  POR  is  fixed  for  each  FY.  PROM-WED  only  reports  these  fixed 
numbers  (i.e.,  they  are  the  same  for  each  run  and  are  not  included  in  the  DOE). 
Each  bar  of  the  stacked  bar  chart  is  divided  into  segments  that  represent  the 
amount  of  resources  allocated  to  each  decision  variable.  A  difference  between  a 
PROM-WED  excursion  and  a  POR  conveys  to  an  analyst  that  the  Navy  should 
consider  allocating  funds  differently  to  optimize  the  allocation  of  recruiting 
resources.  These  insights  support  informed  decisions  such  as  adjusting  the 
number  of  Sailors  assigned  to  recruiting  duty  or  modifying  the  amount  of 
resources  allocated  to  advertisements  and  enlistment  bonuses.  Education 
incentives  were  not  included  in  the  decision  support  analysis,  but  can  be  added  if 
the  Navy  begins  to  allocate  funds  towards  this  resource  again. 

The  scenario  report,  shown  in  Table  10,  reports  the  high  and  low  values  of 
each  market  factor  for  this  PROM-WED  excursion. 


Table  10.  PROM-WED  Scenario  Report 
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0.95 
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0.95 
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0.95 

0.95 

TSC 

high 

0.85 

0.85 
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If  the  high  and  low  values  are  equal,  the  market  factor  is  fixed,  such  as 
NCO  in  the  scenario  shown  in  Table  10.  If  the  market  factor  is  tested  over  a 
range,  the  high  and  low  values  are  not  equal,  such  as  assessing  the  effect  of 
varying  the  percentage  of  high  quality  recruits  (TSC)  from  70  percent  to  85 
percent,  also  shown  in  Table  10. 

The  focus  of  the  second  page  is  on  how  the  decision  variables  vary.  The 
“Total  Cost  of  Recruiting”  stacked  bar  chart  shown  in  Figure  34  indicates  how 
much  money  is  allocated  to  each  recruiting  resource  over  a  seven  FY  span. 


Figure  34.  Graph  3:  Total  Cost  of  Recruiting 
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The  following  three  graphs,  shown  in  Figure  35,  represent  how  deviations 
in  controllable  and  uncontrollable  factors  affect  the  amount  of  resources  allocated 
to  each  decision  variable.  The  blue  dots  represent  the  mean  for  each  decision 
variable  over  each  FY,  and  the  red  dashed  lines  represent  the  95  percent 
confidence  interval  about  that  mean. 


50 


Figure  35.  Graphs  4-6:  Decision  Variables 
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b.  JMP  Output 

PROM-WED  provides  users  with  output  results  that  are  saved  as  an  .xlsx 
file  and  can  be  further  analyzed  using  any  statistical  software  package.  N1 
analysts  use  JMP;  hence  PROM-WED’s  output  is  named  “JMP  output.”  JMP  has 
modeling  tools,  such  as  partition  trees  and  stepwise  regression  models,  that  are 
conducive  for  testing  interactions  between  multiple  variables  while  quantifying 
and  visualizing  how  they  affect  the  overall  solution  space. 

PROM-WED’s  JMP  output  is  color-coded  by  variable  type,  and  is 
organized  for  ease  of  import  into  a  data  analysis  package.  A  snap  shot  of  the 
JMP  output  for  one  FY  of  a  33  design  point  PROM-WED  excursion  is  shown  in 
Figure  36. 
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Figure  36.  JMP  Output  for  a  33  Design  Point 
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Blue  represents  the  output:  Total  Cost  of  Recruiting,  green  represents  the  decision 
variables,  orange  represents  policy  factors,  and  red  represents  the  environmental 
factors. 

This  thesis  uses  JMP  Pro  Version  12  to  analyze  PROM-WED  data  using 
six  primary  techniques:  (1)  oneway  analysis  graphs,  (2)  distributions  and 
descriptive  statistics,  (3)  partition  trees,  (4)  stepwise  regression  models,  (5) 
scatterplot  matrices,  and  (6)  contour  plots  (JMP  Pro,  2015).  The  purpose  of  this 
section  is  to  explain  the  principal  techniques  that  are  used  in  the  analysis  section. 
With  many  of  these  techniques  additional  analysis  could  be  done.  The  analysis 
provided  in  this  research  is  illustrative  of  what  analysts  can  do  with  PROM-WED 
output. 

(1 )  Oneway  Analysis  Graphs 

A  oneway  analysis  graph  is  used  to  gain  a  quantifiable  understanding  of 
the  spread  of  the  total  cost  of  recruiting  data  over  each  FY.  The  setup  and 
structure  of  this  graph  is  shown  in  Figure  37. 
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Figure  37.  Oneway  Analysis  of  Total  Cost  of  Recruiting  by  FY  Structure 
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The  boxplots  that  overlay  the  data  represent  the  information  presented  in 
the  “Quantiles”  table.  From  Figure  37,  it  is  evident  that  more  than  50  percent  of 
the  data  (i.e.,  the  median)  is  less  than  the  grand  mean.  The  grand  mean  is 
represented  by  the  horizontal  line  labeled  “mean,”  and  the  median  is  represented 
by  the  “50%”  label.  The  median  is  a  useful  estimator  that  provides  safety  against 
outliers,  whereas  the  mean  is  highly  influenced  by  extreme  values,  both  high  and 
low. 


(2)  Distributions  and  Descriptive  Statistics 

Histograms  provide  insight  regarding  the  nature  of  the  output  data.  For 
example,  Figure  38  shows  that  the  total  cost  of  recruiting  is  highly  skewed  to  the 
right.  The  long  tail  indicates  that  there  are  some  particularly  large  outliers,  but  the 
majority  of  the  data  does  not  follow  this  trend. 
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Figure  38.  Distribution  and  Descriptive  Statistics  Structure 


»  Total  Cost  of  Recruiting 


H  |o| - f . 

r— 

— | 

300  400  500  600  700  800  900  1000 


Quantiles 

A 

~  Summary  Statistics 

100.0% 

maximum 

974.1696 

Mean 

398.24689 

99.5% 

974.1696 

Std  Dev 

124.63855 

97.5% 

875.44575 

Std  Err  Mean 

10.973812 

90.0% 

533.85 

Upper  95%  Mean 

419.96045 

75.0% 

quartile 

433.12145 

Lower  95%  Mean 

376.53333 

50.0% 

median 

361.2135 

N 

129 

25.0% 

quartile 

320.18825 

10.0% 

293.5959 

2.5% 

276.82095 

0.5% 

261.6095 

0.0% 

minimum 

261.6095 

(3)  Partition  Tree 

The  setup  and  structure  for  a  partition  tree  is  shown  in  Figure  39. 


Figure  39.  JMP  Partition  Tree  Structure 
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A  story  can  be  told  from  interpreting  a  partition  tree.  For  instance,  the  tree 
shown  in  Figure  39  conveys  the  following  message: 

The  mean  total  cost  of  recruiting  will  be  approximately  $354  million.  Since 
advertising  is  the  first  child  of  the  parent  node,  advertising  is  the  dominant 
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decision  variable,  where  66.5  percent  of  the  variance  for  the  total  cost  of 
recruiting  can  be  explained.  If  the  cost  of  advertising  remains  below  $51.4  million, 
then  the  average  cost  of  recruiting  is  approximately  $325  million.  If  the  cost  of 
advertising  equals  to  or  exceeds  $51.4  million,  then  the  average  cost  of  recruiting 
increases  to  $432  million. 

(4)  Stepwise  Regression  Model 

Stepwise  regression  can  be  used  to  formulate  a  prediction  model  for  total 
cost  of  recruiting,  as  shown  in  Figure  40. 


Figure  40.  Stepwise  Regression  Structure 
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The  beta  estimates  and  regression  terms  shown  in  Figure  40  are  used  to 
formulate  the  prediction  model  shown  in  Figure  41. 
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Figure  41 .  Prediction  Model  for  Total  Cost  of  Recruiting  Fit  Using 

Stepwise  Regression 

Prediction  Expression 

150.980052850998 
+  0.02553596931337  *  NCO 
+  -24.593679044673*  Unemployment 
+  -550.34801858165*  Relative  Pay 

+  ( NCO  -  35000.0620155039 )  *  ( ( Relative  Pay  - 1 )  *  -0.1270005027034  ] 

+  ( NCO  -  35000.0620155039 )  *  ( ( NCO  -  35000.0620155039 )  *  0.00000304992539 ) 
+  ( Relative  Pay  - 1 )  *  ( ( Relative  Pay  - 1 )  *  1882.37625427676) 


Actual  by  predicted  plots,  as  shown  in  Figure  42,  demonstrate  the 
relationship  between  the  actual  data  and  the  model  fit  using  stepwise  regression. 
In  this  case,  the  closer  the  points  are  to  the  solid  red  line  the  better  the  fit. 

Figure  42.  Actual  by  Predicted  Plot. 
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(5)  Scatterplot  Matrices 

Each  panel  of  the  scatterplot  matrix  in  Figure  43  shows  the  relationship 
between  a  decision  variable,  on  the  x-axis,  and  the  total  cost  of  recruiting,  on  the 
y-axis. 


Figure  43.  Scatterplot  Matrix  Structure 
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The  dark  red  line  within  the  shaded  red  region  indicates  a  trend  line  fit  in 
JMP.  From  these  scatterplot  matrices,  trends  can  be  deduced  to  help  analysts 
further  understand  the  relationships  amongst  the  model’s  variables.  For  example, 
both  advertising  and  EB  show  a  distinct,  upward  linear  trend  in  relation  to  the 
total  cost  of  recruiting.  The  narrow  confidence  bands  around  the  trend  line  also 
indicate  this  is  a  strong  relationship.  Whereas,  the  total  number  of  recruiters  has 
only  a  minor,  downward  trend.  The  wider  confidence  interval  around  the  trend 
line  for  this  plot  indicates  that  the  total  number  of  recruiters  has  minimal  effect  on 
the  total  cost  of  recruiting  for  this  scenario. 

(6)  Contour  Plots 

Contour  plots  provide  insights  similar  to  the  “capture  the  flag”  example 

previously  shown  in  Figure  22,  where  the  multi-level  full  factorial  DOE  provides  a 

detailed  understanding  of  the  solution  space.  The  contour  plot  in  Figure  44 

shows  the  relation  between  relative  pay  and  accession  mission  on  the  total  cost 
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of  recruiting.  Note  that  other  factors  are  changing  too,  so  it  is  important  to  look  for 
broad  trends,  not  local  features. 


Figure  44.  Contour  Plot  Structure 
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The  color  variations  in  Figure  44  represent  the  total  cost  of  recruiting  at 
different  combinations  of  relative  pay  and  new  accession  mission  values.  The 
diagonal  nature  of  the  plot  indicates  there  is  an  interaction  between  relative  pay 
and  the  new  accession  mission.  To  minimize  the  total  cost  of  recruiting,  it  is 
recommended  that  the  Navy  stays  within  the  dark  blue  regions  if  the  higher 
relative  pay  is  feasible. 

c.  Building  PROM-WED:  Collaboration  with  Future  Users 

To  ensure  the  practicality  and  future  usability  of  this  research,  analysts  at 
N1  played  a  critical  role  in  the  creation  of  the  PROM-WED  tool,  specifically  in 
regards  to  the  GUI  development  and  the  decision  support  capabilities.  A  future 
PROM-WED  user  had  hands-on  time  with  the  tool  to  test  its  limitations  and 
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identify  potential  glitches.  Through  this  meeting,  we  identified  problems  with  the 
save  scenario  capability  and  identified  sources  of  potential  confusion  that  needed 
clarification  and  were  subsequently  addressed  within  the  PROM-WED  User 
Manual.  In  addition  to  the  GUI,  N1  analysts  were  involved  in  the  development  of 
PROM-WED’s  decision  support  capability.  For  instance,  the  JMP  output  color¬ 
coding  and  the  scenario  report  were  added  to  the  automatically  generated 
decision  support  capability  based  on  feedback  from  N1  analysts. 

As  with  any  new  tool,  it  may  take  time  for  N1  analysts  to  become 
accustomed  to  using  PROM-WED.  For  example,  it  was  requested  that  a  graph 
be  added  to  the  automatically  generated  decision  support  capability  that 
displayed  how  unemployment  rate  effects  the  total  cost  of  recruiting  over  each 
FY.  An  example  of  this  graph  is  shown  in  Figure  45.  The  parameter  inputs  for  the 
PROM-WED  excursions  shown  in  Figures  45  and  46  can  be  found  in  Appendix 
D. 


Figure  45.  Effect  of  Unemployment  Rate  on  Total  Cost  of  Recruiting 


380 


—•—2015 

—•—2016 

-•—2017 

—•—2018 

—•—2019 

-•—2020 

—•—2021 


As  expected,  when  the  unemployment  rate  is  low,  the  cost  of  recruiting  is 
high,  and  as  unemployment  rate  increases  the  cost  of  recruiting  decreases. 

However,  PROM-WED  is  capable  of  testing  uncertainties  in  multiple  variables, 
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not  just  one.  When  more  than  one  variable  is  tested  over  a  range,  the  graph 
becomes  difficult  to  interpret.  For  example,  Figure  46  is  a  PROM-WED  excursion 
with  the  same  input  parameters  as  the  PROM-WED  excursion  shown  in  Figure 
45  except  the  number  of  recruiters  is  bounded  from  2,500  to  3,500,  instead  of 
fixed  at  3,913  as  shown  in  Figure  45. 

Figure  46.  Effect  of  Varying  Unemployment  Rate  and  Number  of 
Recruiters  on  Total  Cost  of  Recruiting 
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From  Figure  46,  it  is  evident  that  bounding  the  number  of  recruiters  does 
affect  the  total  cost  of  recruiting.  However,  it  is  difficult  to  discern  any  valuable 
insights  from  Figure  46  regarding  the  interactions  that  are  occurring  between  the 
varied  number  of  recruiters  and  the  unemployment  rate  on  the  total  cost  of 
recruiting.  This  example  only  varied  two  variables,  whereas  excursions  that  are 
explored  in  the  next  section  vary  up  to  six  variables. 

Examples  such  as  this  one  demonstrate  that  through  the  implementation 
of  DOE  techniques,  PROM-WED  delivers  results  that  provide  valuable  insights 
into  the  optimal  allocation  of  recruiting  resources.  However,  this  added  capability 
challenges  the  legacy  analysis  methods  used  to  study  legacy  PRO  model 
outputs. 
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IV.  ANALYSIS 


Through  two  test  case  examples,  this  chapter  showcases  PROM-WED’s 
ability  to  deliver  comprehensive  insights  into  the  optimal  allocation  of  recruiting 
resources.  The  chapter  begins  with  the  introduction  of  the  two  test  case 
examples,  referred  to  as  Test  Case  1  and  Test  Case  2.  These  examples  are  first 
analyzed  through  PROM-WED’s  automatically  generated  decision  support 
capabilities,  and  further  explored  using  an  array  of  statistical  modeling  and 
graphing  methods  in  JMP.  Finally,  a  modified  version  of  Test  Case  1  is  used  to 
compare  the  number  of  runs  required  for  a  full  factorial  DOE  to  the  NOLH 
designs  used  in  PROM-WED. 

A.  TEST  CASES 

To  demonstrate  PROM-WED’s  capabilities,  N1  formulated  three  separate 
scenarios  to  model  best  case,  worst  case,  and  most  likely  situations  for  Navy 
recruiting.  These  scenarios  are  found  in  Appendix  E.  Rather  than  running  three 
separate  scenarios,  PROM-WED  can  test  this  broad  spectrum  of  possibilities  and 
uncertainties  using  a  single  data  farming  run. 

Test  Case  1  explores  uncertainties  in  economically  driven  market  factors 
(i.e.,  relative  pay  and  unemployment  rate).  Test  Case  2  adds  two  additional 
degrees  of  uncertainty  to  Test  Case  1  in  the  form  of  policy  factor  changes  (i.e., 
QMA  and  recruit  quality).  All  market  factors  not  listed  in  the  tables  remain  at  their 
default  values  from  the  legacy  PRO  model.  The  scenario  reports  for  each  run  are 
available  in  Appendix  D. 

a.  Test  Case  1 

Test  Case  1  covers  a  broad  spectrum  of  economic  uncertainties  that 
represent  best  case,  worst  case,  and  most  likely  scenarios  for  Navy  recruiting. 
For  example,  a  low  unemployment  rate,  relative  pay  favoring  the  civilian  sector, 
and  a  high  recruiting  accession  mission  are  challenging  conditions  for  Navy 
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recruiting.  On  the  other  hand,  a  high  unemployment  rate,  relative  pay  favoring 
the  military,  and  a  low  recruiting  accession  mission  would  be  favorable  conditions 
for  Navy  recruiting.  The  input  values  for  Test  Case  1  are  shown  in  Table  1 1  and 
can  be  used  to  answer  a  question  such  as: 

What  is  the  optimal  allocation  of  recruiting  resources  that  is  robust  to  a 
broad  range  of  economic  uncertainties? 


Table  11.  Test  Case  1  Input  Variables 


Variable  Type 

Variable  Name 

Value  Low 

Value  High 

Decision  Variable 

Recruiters 

2,500  recruiters 

3,500  recruiters 

Market  Factor 

Unemployment  Rate 

4.0% 

8.0% 

Market  Factor 

Relative  Pay 

0.80 

1.20 

Policy  Factor 

Recruiting  Mission  (NCO) 

30,000  recruits 

40,000  recruits 

For  additional  scenario  details,  refer  to  Appendix  D. 

b.  Test  Case  2 

Test  Case  2  maintains  the  foundation  of  Test  Case  1 ,  but  adds  the  effects 
of  varying  two  policy  factors:  (1)  percentage  of  high  quality  recruits,  and  (2) 
qualified  military  available.  Test  Case  2’s  input  variables  are  shown  in  Tables  12 
and  13,  and  can  be  used  to  answer  a  question  such  as: 

What  is  the  optimal  allocation  of  recruiting  resources  if  the  Navy  desires  to 
increase  the  percentage  of  high  quality  recruits  from  70  percent  to  85  percent? 
Due  to  uncertainties  in  the  current  fiscal  environment,  the  unemployment  rate 
may  fluctuate  between  4  to  8  percent,  and  the  ratio  of  relative  pay  may  vary 
between  0.8  and  1.2.  In  addition,  since  marijuana  has  been  legalized  for 
recreational  use  in  many  states  nationwide,  drug-use  amongst  18-24  year-olds  is 
expected  to  increase.  An  increase  in  drug-use  amongst  this  age  group  means 
fewer  young  adults  qualify  for  military  service.  Test  Case  2  models  the  effect  of 
an  annual  decrease  of  10,000  qualified  military  available  due  to  pre-service  drug- 
use. 
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Table  12.  Test  Case  2  Input  Variables 


Variable  Type 

Variable  Name 

Value  Low 

dSlillsIKHi 

Decision  Variable 

Production  Recruiters 

2,500  recruiters 

3,500  recruiters 

Market  Factor 

Unemployment  Rate  (UE) 

4.0% 

8.0% 

Market  Factor 

Percentage  of  High  Quality 
Recruits  (TSC  l-lll) 

70% 

85% 

Market  Factor 

Relative  Pay 

0.8 

1.2 

Market  Factor 

Qualified  Military  Available 
(QMA) 

*See  Table  13 

Policy  Factor 

Recruiting  Mission  (NCO) 

30,000  recruits 

40,000  recruits 

Since  Test  Case  2  models  the  cumulative  effects  that  the  legalization  of 
marijuana  may  have  on  the  nation’s  QMA,  the  input  values  for  QMA  will  decrease 
by  10,000  each  FY.  The  QMA  input  values  for  Test  Case  2  are  shown  in  Table 
13. 


Table  13.  Traditional  Run  2  QMA  Input  Values 


FY 

QMA  Value  Low 

QMA  Value  High 

2015 

1,873,304 

1,883,304 

2016 

1,863,304 

1,873,304 

2017 

1,853,304 

1,863,304 

2018 

1,843,304 

1,853,304 

2019 

1,833,304 

1,843,304 

2020 

1,823,304 

1,833,304 

2021 

1,813,304 

1,823,304 

For  more  information  regarding  Test  Case  2  parameter  inputs,  refer  to 
Appendix  D. 

B.  DECISION  SUPPORT  ANALYSIS 

As  explained  in  Chapter  III,  PROM-WED  automatically  generates  a 
selection  of  graphs  to  provide  decision-makers  with  an  “at-a-glance” 
understanding  of  the  solution  space.  The  33-point  design  grows  a  sufficient 
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amount  of  data  for  basic  statistical  analysis  in  under  two  minutes.  Since  the 
purpose  of  the  decision  support  analysis  is  to  provide  a  quick  understanding  of 
the  solution  space,  only  the  33-point  NOLH  design  is  analyzed  in  this  section. 
This  type  of  analysis  would  be  appropriate  for  testing  excursions  during  a  time 
constrained  meeting,  working  group,  or  whenever  basic  analysis  needs  to  be 
generated  quickly.  The  129-point  NOLH  grows  more  data,  requiring  a  longer  run 
time  and  more  time  is  needed  for  adequate  analysis.  The  129-point  NOLH  is 
used  in  the  JMP  analysis  section. 

1.  Test  Case  1 

Some  major  insights  that  are  gained  from  Test  Case  1’s  automatically 
generated  decision  support  capability  are  now  discussed.  Figure  47 
demonstrates  that  in  an  uncertain  economic  environment,  the  mean  total  cost  of 
recruiting  in  FY  2017  will  be  within  $350  million  to  $450  million,  with  95  percent 
confidence. 


Figure  47.  Test  Case  1 :  Mean  Total  Cost  of  Recruiting  over  FYDP 
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Figure  48  indicates  that  on  average,  the  optimal  cost  of  recruiting  for  each 
FY  complements  the  program  of  record  (POR)  budget  estimate. 
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Figure  48.  Test  Case  1 :  PROM-WED  Excursion  versus  POR 
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The  graph  shown  in  Figure  48  can  also  inform  decisions  to  redistribute 
funds  to  optimize  the  allocation  of  resources  to  advertisements,  enlistment 
bonuses,  and  recruiters.  For  example,  in  the  same  graph,  now  labeled  Figure  49, 
informed  recommendations  can  be  made  to  distribute  resources  differently  in 
order  to  optimize  the  allocation  of  recruiting  resources. 


Figure  49.  Test  Case  1 :  PROM-WED  versus  POR 
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Dependent  upon  FY,  if  the  dark  blue  bar  is  higher  for  POR  than  PROM- 
WED,  this  indicates  that  in  order  to  optimize  the  allocation  of  recruiting 
resources,  less  resources  need  to  be  allocated  to  recruiters.  Less  funding 
allocated  to  recruiters  means  less  recruiters  are  required  in  the  field.  The  same 
convention  goes  for  enlistment  bonuses  and  advertisements.  For  example,  in  FY 
2020  less  funds  should  be  allocated  to  recruiting  and  more  funds  should  be 
allocated  to  enlistment  bonuses  and  advertisements. 

Figure  50  shows  that  the  optimal  allocation  of  recruiting  resources 
appears  to  sustain  a  consistent  trend  amongst  the  seven  FYs  with  only  a  minor 
upward  trend,  most  likely  due  to  inflation  rates. 


Figure  50.  Test  Case  1 :  Resource  Allocation  Breakdown 


Insights  gained  through  Figures  48  and  50  indicate  there  is  evidence  to 
believe  that  the  total  cost  of  recruiting  is  robust  to  uncertainties  in  the  economic 
environment.  However,  to  optimize  the  allocation  of  resources,  more  resources 
need  to  be  allocated  to  enlistment  bonuses  and  advertisements,  as  shown 
previously  in  Figure  49. 
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Figure  51  indicates  that,  with  95  percent  confidence,  the  optimal  allocation 
of  resources  to  advertising  over  the  seven  FY  span  is  consistently  maintained 
within  the  range  of  approximately  $40  million  to  $80  million. 


Figure  51 .  Test  Case  1 :  Advertising 
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Similar  to  the  insights  gained  from  Figure  51,  Figure  52  demonstrates  that 
with  95  percent  confidence,  the  optimal  allocation  of  resources  to  enlistment 
bonuses  over  the  seven  FY  span  consistently  maintains  a  range  of  $50  million  to 
$110  million. 


Figure  52.  Test  Case  1 :  Enlistment  Bonuses 
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2. 


Test  Case  2 


The  effects  of  a  shrinking  QMA  pool  and  an  increased  requirement  for 
recruit  quality  is  analyzed  through  the  comparison  of  Test  Case  1  and  Test  Case 
2. 

From  Figure  53,  there  is  evidence  to  believe  that  the  Navy  can  expect  the 
total  cost  of  recruiting  to  increase  by  approximately  $50  million  as  the  need  for 
high  quality  recruits  increases,  and  the  QMA  pool  shrinks.  Without  these  policy 
influences,  the  95  percent  confidence  interval  increased  from  $350  million  to 
$450  million  in  Test  Case  1,  to  approximately  $400  million  to  $500  million  in  Test 
Case  2. 


Figure  53.  Test  Case  2:  Mean  Total  Cost  of  Recruiting  over  FYDP 
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Figure  54  indicates  that  the  total  cost  of  recruiting  is  expected  to  exceed 
the  program  of  record  for  every  FY. 
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Figure  54.  Test  Case  2:  PROM-WED  Excursion  versus  POR 


500 

U  450 
S  400 
2  350 

|  C  300 

O  J  250 


2  200 
“  150 
100 
50 
0 


1 


i  Advertising 
i  EB 

l  Recruiter 
Cost 


O  cc 

O 

CC 

a  cc 

O  CC 

O 

cc 

Q  cc 

O  CC 

LU  O 

UJ 

o 

uj  O 

uj  O 

UJ 

O 

uj  O 

uj  O 

£  °- 

£ 

CL 

$  °- 

£  °- 

Q. 

£  °- 

5 

5 

o 

O 

o 

o 

O 

O 

o 

cc 

cc 

cc 

cc 

cc 

cc 

cc 

Q. 

a. 

Q. 

Q. 

Q. 

o. 

Q. 

2015 

2016 

2017 

2018 

2019 

2020 

2021 

To  optimize  the  allocation  of  recruiting  resources,  there  appears  to  be  a 
consistent  trend  amongst  all  seven  FYs  that  an  excess  of  resources  was 
allocated  to  recruiters  in  the  POR,  while  more  resources  should  be  allocated  to 
advertisements  and  enlistment  bonuses  instead. 

Due  to  the  addition  of  QMA  uncertainties  and  recruit  quality  policy 
changes,  Figure  55  indicates  that  the  average  cost  of  recruiting  is  expected  to 
increase  by  approximately  $50  million  over  the  seven  FY  span.  This  is  a 
noticeable  increase  over  the  trend  previously  shown  for  Test  Case  1  in  Figure  50. 
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Figure  55.  Test  Case  2:  Resource  Allocation  Breakdown 


Figures  56  and  57  juxtapose  results  for  advertisement  and  enlistment 
bonus  resource  allocations  for  Test  Case  1  and  Test  Case  2. 


Figure  56.  Resources  Allocated  to  Advertising 


l/t 

120 

c 

o 

100 

i 

80 

** 

8 

60 

u 

00 

c 

40 

€ 

20 

QJ 

> 

U 

< 

0 

2015  2016  2017 


2018 

FY 


2019  2020  2021 


___  120 
"i/T 

o  100 

ii  so 

B 

» - - 

>  4 

8  60 

... 

00 

c 

2  20 

t : 

dJ  r 

>  u 

5  2015  2016  2017  2018  2019  2020  2021 

FY 


Left:  Test  Case  1 ,  right:  Test  Case  2. 


Figure  56  indicates  that  the  cost  of  advertising  will  increase  by  an  average 
of  approximately  $10  million  each  FY. 
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Figure  57.  Resources  Allocated  to  Enlistment  Bonuses 
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Left:  Test  Case  1 ,  right  Test  Case  2. 

Figure  57  indicates  a  similar  trend  for  enlistment  bonuses.  These  graphs 
show  that  an  additional  $20  million  will  be  required  for  enlistment  bonuses  each 
FY  due  to  the  addition  of  QMA  uncertainties  and  proposed  recruit  quality 
changes. 

C.  GRAPHICAL  AND  STATISTICAL  ANALYSIS  IN  JMP 

Valuable  insights  can  be  found  through  analyzing  variable  interactions  and 
uncertainties  that  shape  the  robust  solution  space.  However,  analyzing  and 
visualizing  variable  interactions  in  Microsoft  Excel  is  difficult  due  to  the  software’s 
limited  statistical  capability.  Analysts  will  need  to  use  a  statistical  software 
package  to  take  full  advantage  of  the  data  grown  by  PROM-WED.  Test  Case  1 
and  Test  Case  2  are  now  analyzed  using  JMP. 

1.  Test  Case  1 

To  gain  an  initial  understanding  of  the  data,  Figure  58  shows  the  spread  of 
data  and  provides  quantile  metrics  for  each  FY.  From  Figure  58,  it  is  evident  that 
over  50  percent  of  the  data,  indicated  by  the  median,  generated  for  each  FY  is 
below  the  grand  mean  total  cost  of  recruiting  for  each  FY. 
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Figure  58.  Total  Recruiting  By  FY  with  Interquartile  Ranges 
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The  outliers  in  Figure  58,  highlighted  below  in  Figure  59,  are  worth 
examining  further  to  determine  if  there  is  a  common  cause  for  the  four  unusual 
data  points.  Using  JMP,  the  highlighted  sixteen  data  points  are  lassoed  (i.e., 
selected)  to  reveal  that  the  78th,  80th,  88th,  and  96th  runs  for  each  scenario 
caused  these  results  over  each  FY.  The  run  numbers  represent  four  of  the  129 
different  scenarios  built  using  PROM-WED’s  129-point  NOLH  DOE.  Since  each 
FY  uses  the  same  NOLH  DOE,  the  80th  run  for  each  FY  of  Test  Case  1  is  tested 
over  the  same  input  market  factors  and  number  of  recruiters.  The  same 
convention  applies  for  the  78th,  88th,  and  96th  runs  as  well. 
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Figure  59.  Outliers  for  Each  FY 
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The  upward  trend  occurs  due  to  yearly  changes,  such  as  inflation  rates, 
elasticities,  or  input  values  from  the  legacy  PRO  model.  The  input  variables  for 
each  run  highlighted  in  Figure  59  are  shown  in  Table  14. 

Table  14.  Test  Case  1  Input  Variables  for  Output  Outliers 


Run  # 


78 

80 

88 

96 

Recruiters 

2547 

2523 

2727 

2789 

NCO 

39531 

39688 

39453 

39609 

UE 

7.0 

5.9 

5.1 

5.1 

Relative  Pay 

0.84375 

0.878125 

0.85 

0.853125 

The  most  extreme  total  cost  of  recruiting  outlier,  resulting  from  the  80th 
run,  tested  an  excursion  where  the  Navy  had  a  very  low  number  of  recruiters  in 
the  field  (just  over  2,500  recruiters),  the  new  accession  mission  was  extremely 
high  (almost  at  40,000  new  recruits),  unemployment  rate  was  mediocre,  and  the 
relative  pay  favored  the  civilian  sector.  The  other  three  runs  also  showed  similar 
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trends  where  high  accession  missions,  with  a  low  number  of  recruiters  in  the 
field,  and  relative  pay  highly  favoring  the  civilian  sector  resulted  in  unusually  high 
expected  recruiting  costs.  Identifying  these  costly  outliers  can  help  N1  analysts 
make  informed  recommendations  to  avoid  situations  like  the  80th  run  by 
preemptively  increasing  the  number  of  recruiters  in  the  field. 

To  gain  additional  situational  awareness  of  the  data,  the  distributions  and 
descriptive  statistics  for  each  decision  variable  are  explored.  Figure  60  shows  the 
histogram  of  the  distribution  for  total  cost  of  recruiting  over  one  FY  of  the  PROM- 
WED  excursion.  Histogram  and  descriptive  statistics  for  resourcing  to 
advertisements  and  enlistment  bonuses  can  be  found  in  Appendix  F. 


Figure  60.  Histogram  and  Descriptive  Statistics  for  Total  Cost  of 

Recruiting  Distribution  for  FY  2017 
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The  histogram  indicates  that  the  distribution  is  highly  skewed  to  the  right. 
As  well,  the  four  data  points  that  appear  in  the  far  right  side  of  the  histogram 
again  represent  runs  78,  80,  88,  and  96. 

Partition  trees  were  used  to  understand  how  variable  interactions  and 
economic  uncertainties  affected  the  solution  space.  The  partition  trees  in  Figures 
61  and  62  take  into  consideration  the  influence  of  each  decision  variable  on  the 
total  cost  of  recruiting.  Figure  61  shows  the  first  split  of  the  partition  tree  for  total 
cost  of  recruiting,  specifically  for  FY  2017. 
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Figure  61 .  First  Split  of  Total  Cost  of  Recruiting  Partition  T ree 


Figure  61  indicates  that  resourcing  to  advertising  is  the  most  influential 
predictor  of  the  total  cost  of  recruiting.  61.9  percent  of  the  variance  for  the  total 
cost  of  recruiting  can  be  explained  based  on  the  first  split  of  the  partition  tree. 
When  less  than  $139  million  is  allocated  to  advertising,  then  the  mean  total  cost 
of  recruiting  will  be  approximately  $371  million.  If  more  than  $139  million  is 
allocated  to  advertising,  then  the  mean  total  cost  of  recruiting  will  increase  to 
almost  $755  million. 

Figure  62  shows  the  next  split  of  the  partition  tree  shown  in  Figure  61 . 
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Figure  62.  Second  Split  of  Total  Cost  of  Recruiting  Partition  Tree 
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The  second  split  of  the  partition  tree  indicates  that  resourcing  to 
advertising  is  identified  again  as  the  dominant  predictor  of  the  total  cost  of 
recruiting.  Based  on  this  split,  over  80  percent  of  variance  in  the  total  cost  of 
recruiting  is  explained.  Repeated  splitting  of  the  same  factor,  in  this  case 
resources  allocated  to  advertising,  indicates  regression  may  be  a  more 
informative  analysis  technique. 

Next,  a  partition  tree  is  used  to  understand  which  market  factors  most 
influence  advertising.  Figure  63  shows  the  parent  and  first  child  node  of  the 
partition  tree  for  advertising. 
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Figure  63.  Parent  and  First  Child  Node  of  Partition  Tree  for  Advertising 


The  partition  tree  in  Figure  63  indicates  that  the  recruiting  accession 
mission  is  the  most  influential  factor  on  the  cost  of  advertising.  The  relatively 
small  R-squared  value  indicates  that  a  single  split  on  accession  mission  explains 
only  39.6  percent  of  variance.  In  particular,  if  the  accession  mission  is  below 
38,750  new  recruits,  the  mean  resourcing  towards  advertising  is  approximately 
$44.4  million.  If  the  accession  mission  exceeds  38,750  new  recruits,  then  the 
mean  resourcing  to  advertising  increases  by  over  $100  million,  to  $148.6  million. 

Following  seven  additional  splits,  as  shown  in  Figure  64,  it  is  evident  that 
the  resourcing  of  funds  to  advertising  is  influenced  by  many  factors,  to  include: 
the  new  accession  mission,  relative  pay,  and  to  a  small  extent,  the 
unemployment  rate.  Since  it  took  seven  splits  to  surpass  the  80  percent  R- 
squared  threshold,  it  is  evident  that  these  three  factors  influence  the  resourcing 
of  funds  to  advertising,  but  none  of  them  particularly  dominate. 
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Figure  64.  Partition  Tree  for  Advertising  After  Seven  Splits 
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Stepwise  regression  is  another  method  used  to  gain  insights  into  how 
variables  influence  the  solution  space.  Using  stepwise  regression  with  some 
manual  judgement,  the  parameter  estimates  shown  in  Figure  65  are  used  to 
formulate  the  prediction  model  for  the  total  cost  of  recruiting,  shown  in  Figure  66. 


Figure  65.  Stepwise  Regression  for  Total  Cost  of  Recruiting 


Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

A 

-O 

o 

L_ 

CL 

Intercept 

150.98005 

70.71931 

2.13 

0.0348* 

NCO 

0.025536 

0.001535 

16.64 

<.0001* 

Unemployment 

-24.59368 

3.836157 

-6.41 

<.0001* 

Relative  Pay 

-550348 

38.3648 

-14.35 

<.0001* 

(NCO-35000.1)*(Relative  Pay-1) 

-0.127001 

0.015311 

-8.29 

<.0001* 

(NCO-35000.1)*(NCO-35000.1) 

3.0499e-6 

6.205e-7 

4.92 

<.0001* 

(Relative  Pay-l)*(Relative  Pay-1) 

1882.3763 

384.6306 

4.89 

<.0001* 

The  stepwise  regression  model  exhibits  how  the  NOLH  DOE  allows  for 
non-linear  relationships  and  interactions. 
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Figure  66.  Test  Case  1 ,  Prediction  model  for  Total  Cost  of  Recruiting 

Prediction  Expression 

150.980052850998 
+  0.02553596931337*  NCO 
+  -24.593679044673*  Unemployment 
+  -550.34801858165*  Relative  Pay 

+  (  NCO -35000.0620155039)*  ((Relative Pay  -l)* -0.1270005027034] 

+  (  NCO  -  35000.0620155039 )  *  ( ( NCO  -  35000.0620155039 )  *  0.00000304992539 ) 
+  ( Relative  Pay  - 1 )  *  ( ( Relative  Pay  - 1 )  *  1882.37625427676 ) 


The  prediction  model  for  total  cost  of  recruiting  indicates  that  the  new 
accession  mission  and  relative  pay  interact  to  effect  the  total  cost  of  recruiting. 
The  new  accession  mission  and  relative  pay  both  exhibit  a  non-linear  behavior  as 
evidence  by  their  polynomial  to  degree  two  interactions.  This  relationship  can 
also  be  visualized  in  the  prediction  profiler  shown  in  Figure  67. 


Figure  67.  Prediction  Profiler  for  Varying  Factors  in  Test  Case  1 
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35000  6.0031  Relative 

NCO  Unemployment  Pay 


The  prediction  profiles  for  the  new  accession  mission  and  relative  pay 
shown  in  Figure  67  demonstrate  their  quadratic  nature. 


79 


Next,  the  summary  of  fit  for  the  regression  model  shown  is  shown  in 
Figure  68. 


Figure  68.  Summary  of  Fit  for  Total  Cost  of  Recruiting  Prediction  Model 
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This  model  explains  over  84  percent  of  the  variance  of  the  total  cost  of 
recruiting  for  FY  2017. 

To  visualize  a  comparison  of  this  model  to  actual  FY  2017  data,  the  actual 
versus  predicted  plot  is  shown  in  Figure  69. 


Figure  69.  Test  Casel ,  Actual  by  Predicted  Plot  for  Total  Cost  of 

Recruiting 
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Figure  70  highlights  the  outlying  points.  Once  again,  runs  78,  80,  88  and 
96  appear  to  be  outliers. 
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Figure  70.  Test  Casel ,  Actual  by  Predicted  Plot  with  Outliers 


Actual  by  Predicted  Plot 


Total  Cost  of  Recruiting  Predicted  P<.0001 
RSq=0.84  RMSE=50.706 

Since  the  new  accession  mission,  unemployment  rate,  and  relative  pay 
drive  advertising  resourcing,  six  scatterplot  matrices,  shown  in  Figure  71,  help 
analysts  visualize  trends  amongst  these  factors  against  the  total  cost  of  recruiting 
and  the  resourcing  of  funds  to  advertising.  As  before,  we  plot  the  response  (in 
this  case,  total  cost  of  recruiting  and  advertising  costs)  against  new  accession 
mission,  unemployment  rate,  and  relative  pay.  The  values  for  the  new  accession 
mission,  unemployment  rate,  and  relative  pay  come  from  the  NOLH  DOE.  Other 
factors  such  as  allocated  funds  to  EB  and  the  number  of  recruiters  in  the  field, 
are  also  changing  (EB  is  being  optimized,  while  number  of  recruiters  comes  from 
the  NOLFI  DOE).  Therefore,  the  trends  in  these  scatterplot  matrices  should  be 
considered  through  the  lens  of  a  broad  picture,  not  localized  trends. 
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Figure  71 .  Economic  Factor  Trends  on  Recruiting  Resource  Allocation 


»  Scatterplot  Matrix 


The  two  scatterplot  matrices  for  NCO  versus  advertising  and  NCO  versus 
total  cost  of  recruiting  both  indicate  an  upward  trend,  where  a  higher  accession 
mission  correlates  with  more  resources  allocated  toward  advertising  and  a  higher 
total  cost  of  recruiting.  Both  of  the  unemployment  rate  graphs  show  minor  signs 
of  a  downward  trend  indicating  that  the  cost  of  recruiting  and  the  allocation  of 
resources  to  advertising  decreases,  as  the  unemployment  rate  increases.  Lastly, 
the  relative  pay  versus  advertising  and  relative  pay  versus  total  cost  of  recruiting 
graphs  also  indicate  a  trend.  As  the  relative  pay  begins  to  increase,  meaning 
wages  favor  the  military  over  the  civilian  sector,  resourcing  towards  advertising 
begins  to  decrease  and  the  total  cost  of  recruiting  also  decreases. 

The  four  outlying  points  from  runs  78,  80,  88  and  96  are  present  in  these 
scatterplots  as  well.  Figure  72  highlights  results  from  these  four  runs.  Once 
again,  they  appear  to  be  outliers  within  each  scatterplot. 
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Figure  72.  Economic  Factor  Trends  on  Recruiting  Resource  Allocation 

with  Outliers 


»  Scatterplot  Matrix 
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Variable  interactions  can  also  be  shown  in  a  three-dimensional  manner 
using  contour  plots.  The  contour  plot  in  Figure  73  represents  the  interaction 
between  relative  pay  and  accession  mission  on  the  total  cost  of  recruiting. 
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Figure  73.  Three-Dimensional  Representation  of  Relative  Pay  and  NCO 

Effects  on  the  Total  Cost  of  Recruiting 


▼  Contour  Plot  for  Total  Cost  of  Recruiting 
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The  diagonal  color  transition  indicates  the  presence  of  interactions.  The 
red  region,  in  the  upper  left  portion  of  the  plot  represents  the  interaction  between 
relative  pay  and  new  accession  mission  that  result  in  the  most  costly  conditions 
for  Navy  recruiting.  This  region  represents  when  wages  favor  the  civilian  sector 
and  the  accession  mission  is  high.  The  dark  blue  area  represents  the  opposite 
conditions,  where  the  total  cost  of  recruiting  is  the  lowest  when  the  accession 
mission  is  relatively  low  and  relative  pay  favors  the  military. 

The  contour  plot  shown  in  Figure  74  illustrates  the  relationship  between 
relative  pay  and  recruit  accession  mission  on  resources  allocated  toward 
advertising. 
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Figure  74.  Three-Dimensional  Representation  of  Relative  Pay  and  NCO 

Effects  Resourcing  to  Advertising 
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Figure  74,  which  also  exhibits  a  diagonal  nature,  indicates  that  nearly  half 
of  the  solution  space  supports  a  low  advertising  budget,  represented  by  the  dark 
blue  region.  The  cost  of  advertising  substantially  increases  when  relative  pay 
favors  the  civilian  sector  and  the  accession  mission  is  high,  represented  by  the 
red  region.  Once  relative  pay  exceeds  approximately  1.00,  changes  in  the  new 
accession  mission  have  little  to  no  effect  on  the  amount  of  resources  allocated  to 
advertising. 


2. 


Test  Case  2 


To  further  understand  how  the  addition  of  two  policy  uncertainties  affect 
the  optimal  allocation  of  recruiting  resources,  Test  Case  2  is  explored  using  JMP. 
As  in  the  previous  section,  emphasis  is  placed  on  comparing  insights  gained  that 
may  distinguish  Test  Case  2  from  Test  Case  1 . 
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To  gain  an  initial  understanding  of  the  data,  Figure  75  shows  the  span  of 
possible  costs  of  recruiting  over  each  FY. 


Figure  75.  Total  Cost  of  Recruiting  by  FY  with  Interquartile  Ranges 


▼  Oneway  Analysis  of  T otal  Cost  of  Recruiting  By  FY 
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It  is  evident  that  the  grand  mean  total  cost  of  recruiting  increased  by 
almost  $50  million  in  comparison  to  Test  Case  1’s  grand  mean  total  cost  of 
recruiting  shown  previously  in  Figure  58.  As  well,  Figure  76  shows  that  runs  80, 
88,  and  96  model  conditions  result  in  unusually  high  expected  recruiting  costs. 
From  Figure  76,  it  is  difficult  to  distinguish  the  difference  between  runs  80  and 
88. 
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Figure  76.  Test  Case  2  Outliers 


▼  Oneway  Analysis  of  T otal  Cost  of  Recruiting  By  FY 
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The  input  values  that  are  common  for  each  run  over  the  seven  FY 
excursion  are  shown  in  Table  15.  The  annual  decrease  in  QMA  values  is 
explored  later  in  this  section. 


Table  15.  Test  Case  2  Outlier  Input  Values 


Run  # 


80 

88 

96 

Recruiters 

2523 

2727 

2789 

NCO 

39688 

39453 

39609 

UE 

5.9 

5.1 

5.1 

Relative  Pay 

0.878125 

0.85 

0.853125 

TSC  l-IIIA 

0.83 

0.71 

0.85 

In  comparison  to  Test  Case  1,  where  the  96th  run  was  the  “least  extreme 
of  the  extreme”  values,  the  96th  run  for  Test  Case  2  consistently  modeled  the 
“most  extreme  of  the  extreme”  values.  This  indicates  that  the  increase  in  recruit 
quality  and  annual  decrease  in  QMA  affected  the  optimal  allocation  of  recruiting 


resources. 
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Additional  insights  can  be  gained  by  comparing  the  quantile  metrics  for 
both  Test  Cases.  The  quantile  charts  for  both  test  cases  are  shown  in  Figure  77. 


Figure  77.  Test  Case  1  and  Test  Case  2  Quantile  Charts 


<4  Quantiles 

Level 

Minimum 

10% 

25% 

Median 

75% 

90% 

Maximum 

2015 

251.8643 

283.3929 

309.1051 

349.9401 

421.2093 

521.9235 

959.1079 

2016 

258.138 

289.8711 

315.9455 

356.89 

428.4688 

529.2777 

968.0586 

2017 

261.6095 

293.5959 

320.1883 

361.2135 

433.1215 

533.85 

974.1696 

2018 

265.1709 

297.4278 

324.5511 

365.861 

437.8086 

538.5172 

980.3789 

2019 

268.7332 

301.3375 

329.0443 

370.5412 

442.3917 

543.1901 

986.5974 

2020 

272.8857 

305.6795 

334.2964 

375.5285 

447.5582 

548.4681 

993.3847 

2021 

277.2888 

310.3785 

339.8189 

380.7692 

453.0851 

554.0022 

100043 

^  Quantiles 

Level 

Minimum 

10% 

25% 

Median 

75% 

90% 

Maximum 

2015 

253.7634 

296.5226 

318.9388 

378.5047 

467.3705 

571.5925 

1019.142 

2016 

260.1305 

303.2767 

325.8074 

385.8551 

475.3449 

580.2986 

1030.877 

2017 

263.6967 

307.3261 

329.9298 

390.532 2 

480.6594 

586.3599 

1039.902 

2018 

267.3544 

311.4732 

334.3055 

395.3136 

486.0845 

592.5386 

1049.072 

2019 

271.0144 

315.6291 

338.7466 

400.1104 

491.531 

598.7463 

1058.296 

2020 

275266 

320347 

343.8861 

405.5857 

497.6674 

605.6559 

1068.192 

2021 

279.7699 

325.1176 

349.2917 

411.3294 

504.0784 

6  12.8481 

1078.395 

Top:  Test  Case  1 ;  bottom:  Test  Case  2. 


Figure  77  helps  inform  analysts  that  over  each  FY,  Test  Case  2  requires 
more  resources  than  Test  Case  1.  The  differences  between  the  minimum  values 
for  each  Test  Case  are  approximately  $2  million  across  each  FY.  This  spread 
can  increase  upwards  of  $70  million  when  comparing  differences  between 
maximum  values  of  both  cases.  As  well,  the  interquartile  ranges,  the  difference 
between  the  25th  and  75th  quartiles  which  represent  50  percent  of  the  data,  is 
approximately  $113  million  for  Test  Case  1  and  increases  to  approximately  $150 
million  for  Test  Case  2. 

Figure  78  juxtaposes  the  distributions  and  descriptive  statistics  for  Test 
Case  1  and  Test  Case  2. 
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Figure  78.  Distributions  and  Descriptive  Statistics  for  Test  Cases  1  and 
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Top:  Test  Case  1 ;  bottom:  Test  Case  2. 


Figure  78  indicates  that  the  distribution  of  recruiting  costs  for  Test  Case  2 
is  positively  skewed  with  a  long  right  tail,  as  was  the  case  for  Test  Case  1.  Test 
Case  2’s  right  tail  appears  to  be  wider  than  what  was  seen  for  Test  Case  1.  A 
wider  right  tail  indicates  that  Test  Case  2  produced  more  expensive  combinations 
of  recruiting  resources,  also  referred  to  as  outliers,  in  comparison  to  Test  Case  1. 

When  comparing  the  mean  and  median  values  for  each  Test  Case,  the 
differences  between  the  mean  and  median  values  for  Test  Case  1  and  Test  Case 
2  are  approximately  equal,  at  $37  million  and  $38  million,  respectively.  This 
suggests  that  the  mean  total  cost  of  recruiting  is  heavily  influenced  by  the 
outliers,  but  even  with  the  presence  of  more  outliers  in  Test  Case  2,  the 
differences  between  the  mean  and  median  estimators  are  negligible. 
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As  in  Test  Case  1,  a  partition  tree  identifies  advertising  as  the  most 
influential  decision  variable  for  Test  Case  2.  Figure  79  shows  that  62.1  percent  of 
variance  in  the  total  cost  of  recruiting  can  be  explained  from  a  split  on 
advertising. 


Figure  79.  First  Split  for  Test  Case  2 


▼  Partition  for  Total  Cost  of  Recruiting 


Following  four  splits,  Figure  80  indicates  that  when  the  R-squared  value 
exceeds  .80,  and  even  .92  in  this  case,  advertising  continues  to  dominate  the 
partition  tree. 
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Figure  80.  Test  Case  2  Following  Four  Splits 


▼  Partition  for  Total  Cost  of  Recruiting 
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As  in  Test  Case  1,  repeated  splitting  on  advertising  indicates  regression 
as  an  appropriate  technique  for  further  analysis. 

Partition  trees  are  also  constructed  to  determine  how  uncertainties  in  QMA 
and  a  policy  change  in  recruit  quality  could  affect  resourcing  to  advertising.  Here 
advertising  is  the  response  variable  and  we  are  investigating  which  factors 
influence  advertising.  Figure  81  shows  the  first  split  of  this  tree. 
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Figure  81 .  Test  Case  2:  First  Split  of  Advertising 


▼  Partition  for  Advertising 


Similar  to  results  found  in  Test  Case  1,  the  new  accession  mission  is 
identified  as  the  dominant  factor,  but  it  maintains  a  low  variance  explained  at 
37.8  percent.  Following  three  more  splits,  the  R-squared  value  doubled.  The 
resulting  partition  tree  is  shown  in  Figure  82. 
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Figure  82.  Test  Case  2:  Partition  Tree  for  Advertising 


▼  Partition  for  Advertising 
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56.721447  16.796722  48.7264 

38.388607 

Relative  Pay  >=0.953125 

Count  72  LogWorth  Difference 

Mean  39.319154  14.051149  32.69 

Std  Dev  23.047752 

Relative  Pay<0. 953125 
Count  40 

Mean  88.045575 

Std  Dev  40.765568 

b  Candidates 

Relative  Pay>=0.9125 
Count  11 

Mean  108.46275 
Std  Dev  35.890963 

Relative  Pay  <0.9125 
Count  6 

Mean  271.0387 

Std  Dev  46.293454 

>  Candidates 

>  Candidates 

NCO<36250 

Count  49 

Mean  28.876531 
Std  Dev  12.654269 

NCO>=36250 

Count  23 

Mean  61.566483 
Std  Dev  24.589152 

t  Candidates 

t  Candidates 

Figure  82  indicates  that  the  new  accession  mission  and  relative  pay 
predominately  drive  the  allocation  of  resources  to  advertising.  It  is  interesting  to 
note  that  neither  QMA  nor  recruit  quality  appear  in  this  partition  tree.  This 
suggests  that  they  have  a  minimal,  if  any,  influence  on  advertising  resources. 

Again,  stepwise  regression  with  manual  judgement,  is  used  to  formulate  a 
model  to  predict  the  total  cost  of  recruiting  for  Test  Case  2.  The  parameter 
estimates  used  to  formulate  the  prediction  model  are  shown  in  Figure  83. 
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Figure  83.  Test  Case  2,  Stepwise  Regression  for  Total  Cost  of 

Recruiting 


Parameter  Estimates 

Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept 


NCO 
TSCI-EIA 
Unemployment 
Relative  Pay 

(NCO-35000.1)*(Unemployment-6.0031) 
(NCO-35000.1)*(Relative  Pay-1) 

(TSC  I-IIIA-0.77  504)*(  Relative  Pay-1) 
(Unemployment-6. 0031)*(Relative  Pay-1) 
(NCO-35000.1)*(NCO-35000.1) 

(Relative  Pay-l)*(Relative  Pay-1) 


-126.5045 

74.71878 

-1.69 

0.0931 

0.0266897 

0.001086 

24.57 

<.ooor 

562.69782 

72.19942 

7.79 

<.ooor 

-33.33182 

2.715898 

-1227 

<.ooor 

-662.512 

27.15714 

-2440 

<.ooor 

-0.005165 

0.001003 

-5.15 

<.0001 

-0.120308 

0.010921 

-11.02 

<.0001 

-2866.773 

673.4763 

-4.26 

<.0001 

135.38315 

24.1139 

5.61 

<.0001 

2.3921e-6 

4.485e-7 

5.33 

<.0001 

1980.2703 

278.714 

7.11 

<.0001’ 

The  parameter  estimates  shown  in  Figure  83,  formulate  the  prediction 
model  shown  in  Figure  84. 


Figure  84.  Test  Case  2,  Prediction  model  for  Total  Cost  of  Recruiting 

Prediction  Expression 

-126.5044997643 
+  0.02668971455148  *  NCO 
+  562.697816223402  *  TSC  I-IDA 
+  -33.3318158052  *  Unemployment 
+  -662.51203008942  *  Relative  Pay 

+  (  NCO  -  35000.0620155039 )  *  ( ( Unemployment  -  6.0031007751938  ]  *  -0.0051646636056 ) 
+  (  NCO-  35000.0620155039)  *((  Relative  Pay  -l)*  -0.1203081976383) 

+  (TSCI-mA  -  0.77503875968992)  *  ((RelativePay  - 1)  * -2866.7729142712) 

+  ( Unemployment  -  6.0031007751938)*  ((RelativePay  - 1)  *  135.383146993767) 

+  (  NCO  -  35000.0620155039 )  *  ( ( NCO  -  35000.0620155039 )  *  0.00000239213948 ) 

+  (  Relative  Pay  - 1 )  *  ( (  Relative  Pay  - 1 )  *  1980.27034394266  ) 
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As  in  Test  Case  1,  this  prediction  model  indicates  the  presence  of  variable 
interactions  and  non-linear  effects.  Test  Case  2  appears  to  be  highly  influenced 
by  multi-variable  interactions.  Where  Test  Case  1  had  just  one  multi-variable 
interaction  and  two  quadratic  terms,  Test  Case  2’s  prediction  model  has  four 
multi-variable  interactions  and  two  quadratic  terms.  The  regression  model  shown 
in  Figure  84  provides  evidence  to  believe  that  the  addition  of  these  two  policy 
uncertainties  (i.e.,  percentage  of  high  quality  recruits  and  decrease  in  QMA)  does 
increase  the  complexity  of  recruiting  resource  allocation  and  effects  the  total  cost 
of  recruiting. 

The  summary  of  fit  for  Test  Case  2’s  prediction  model  is  shown  in  Figure 
85. 


Figure  85.  Summary  of  Fit  for  Test  Case  2’s  Prediction  Model  for  Total 

Cost  of  Recruiting 


Summary  of  Fit 

RSquare 

0.935761 

RSquare  Adj 

0.930318 

Root  Mean  Square  Error 

35.89242 

Mean  of  Response 

428.3678 

Observations  [or  Sum  Wgts) 

129 

This  prediction  model  explains  over  93  percent  of  the  variance  in  the  total 
cost  of  recruiting. 

The  actual  versus  predicted  plot  in  Figure  86  illustrates  how  the  prediction 
model  compares  to  the  actual  data  for  FY  2017. 
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Figure  86.  Test  Case  2,  Actual  versus  Predicted  Plot 


Actual  by  Predicted  Plot 


300  400  500  600  700  800  900  1000 
Total  Cost  of  Recruiting  Predicted  P<.0001 
RSq=0.94  RMSE=35.892 


The  factors  that  were  determined  to  be  influential  through  the  partition  tree 
and  stepwise  regression  are  fit  in  scatterplot  matrices  to  visualize  trends  or 
relationships  of  the  data,  as  shown  in  Figure  87.  QMA  was  also  included  for 
comparison  even  though  it  is  not  considered  an  influential  factor. 


Figure  87.  Test  Case  2:  Scatterplot  Matrices  of  Influential  Factors 
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As  in  Test  Case  1,  relative  pay  and  NCO  follow  similar  trends.  Both 
scatterplot  graphs  for  the  percentage  of  high  quality  recruits  show  a  slightly 
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upward  linear  trend.  This  indicates  that  an  increased  percentage  of  high  quality 
recruits  requires  more  resourcing  to  advertising,  thus  resulting  in  high  overall 
recruiting  costs.  Both  unemployment  rate  graphs  show  a  slight  downward  trend, 
indicating  that  the  total  cost  of  recruiting  and  the  total  cost  of  advertising 
decreases  as  the  unemployment  rate  increases.  The  recruit  quality  scatterplots 
suggest  that  as  the  requirement  for  recruit  quality  increases,  more  funds  need  to 
be  allocated  to  advertising  and  the  total  cost  of  recruiting  increases.  Both 
scatterplots  for  QMA  do  not  indicate  any  discernible  trends. 

D.  FULL  FACTORIAL  COMPARISON 

The  NOLH  DOE  technique  is  the  foundation  for  PROM-WED’s  data 
farming  wrapper.  Coupled  with  PROM-WED’s  GUI,  users  are  able  to  design, 
populate,  and  execute  space-filling  experimental  designs  quickly  and  easily. 
Without  the  NOLH  DOE,  PROM-WED’s  data  farming  wrapper  would  not  be  as 
effective. 

As  previously  described  in  Chapter  II,  the  NOLH  DOE  method  is  an 
alternative  to  the  straightforward  full  factorial  method.  A  modified  version  of  Test 
Case  1  is  used  to  demonstrate  what  a  potential  full  factorial  could  look  like.  This 
design  tests  three  variables  at  only  nine  levels  each.  Table  16  shows  an 
illustrative  example  of  what  nine  levels  for  each  variable  could  look  like. 


Table  16.  Full  Factorial  Levels  for  Modified  Test  Case  1 


Levels 

Relative  Pay 

Unemployment  Rate 

Recruiters 

1 

0.80 

4.0% 

30,000 

2 

0.85 

4.5% 

31,000 

3 

0.90 

5.0% 

32,000 

4 

0.95 

5.5% 

33,000 

5 

1.00 

6.0% 

34,000 

6 

1.05 

6.5% 

35,000 

7 

1.10 

7.0% 

36,000 

8 

1.15 

7.5% 

37,000 

9 

1.20 

8.0% 

38,000 
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In  comparison  to  the  NOLH  DOE,  where  each  variable  is  tested  at  either 
33  or  129  levels,  for  this  full  factorial  example  each  variable  is  tested  over  only 
nine  levels.  To  test  all  possible  variable  interactions  the  full  factorial  DOE  would 
have  to  be  run  over  729  input  combinations  for  each  FY.  729  runs  for  each  FY 
results  in  5,103  runs  for  all  seven  FY’s.  This  is  in  comparison  to  231  runs  for  the 
33-point  NOLH  design,  or  903  runs  for  the  129-point  NOLH  design,  which 
account  for  all  runs  over  all  seven  FYs.  The  pairwise  scatterplot  matrices  of  a 
multi-level  full  factorial  design  in  comparison  to  the  129  design  point  NOLH  are 
shown  in  Figure  88. 


Figure  88.  Pairwise  Plots  of  Full  Factorial  versus  NOLH  129  Point 

Designs 


Recruiters  Pay  Rate  Recruiters  Pay  Rate 

Left:  Full  Factorial.  Right:  128-point  NOLH  DOE. 


As  is  evident  by  these  pairwise  plots,  the  NOLH  DOE  is  able  to  execute 
space-filling  designs  with  a  fraction  of  runs. 

Not  only  is  the  NOLH  DOE  method  an  efficient  and  effective  alternative  to 
the  factorial  DOE  method,  PROM-WED  demonstrates  that  the  NOLH  DOE  can 
be  embedded  into  a  model  to  add  a  robust  data  farming  capability.  The  NOLH 
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DOE  algorithm  built  in  Microsoft  Excel  by  the  SEED  Center  for  Data  Farming  at 
NPS  provides  this  capability.  Statistical  software  packages,  like  JMP,  have  a 
factorial  DOE  capability.  However,  to  use  this  method  an  analyst  would  have  to 
build  the  factorial  DOE  in  JMP  and  import  the  design  into  Microsoft  Excel. 
Embedding  the  NOLH  DOE  within  the  legacy  PRO  model  alleviates  this  extra 
step,  while  also  providing  analysts  with  enhanced  analytic  abilities  through 
efficient  and  effective  space-filling  designs  that  provide  opportunities  for  robust 
sensitivity  and  risk  analysis. 

E.  DISCUSSION 

PROM-WED  is  an  enhanced  analytic  tool  capable  of  providing  PRO  model 
users  with  insights  to  better  inform  recruiting  resource  allocation  decisions.  The 
legacy  PRO  model  produces  a  point-solution  output,  as  shown  in  Figure  89. 


Figure  89.  Legacy  PRO  Model  Output 


Resource  Run 

2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCO 

35,025 

36,425 

36,800 

35,800 

35,225 

34,650 

34,650 

Capacity 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Unemployment  (%) 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 

Total  Recruiters 

3,913 

3,685 

3,685 

3,685 

3,685 

3,685 

3,685 

Total  Recruiter  Cost  <$M) 

$320,488 

$305,122 

$309  699 

$314,344 

$319,059 

$324  618 

$330,274 

Advertising  ($M) 

$102,921 

$264,167 

$261  119 

$184,724 

$142  227 

$115,543 

S1 1 3.903 

Enlistment  Bonus  (SM) 

$40  971 

$36  580 

$41  340 

$40,650 

$42,230 

$42  060 

$42  810 

Education  Incentives  ($M) 

$0  000 

$0,000 

$0,000 

$0,000 

$0,000 

$0,000 

$0,000 

LRP  ($M) 

$7  440 

$11,220 

$11  280 

$11,380 

$11,430 

$1 1 .460 

$11,670 

HSDG 

95% 

95% 

95% 

95% 

95% 

95% 

95% 

TSC  l-IIIA 

70% 

70% 

70% 

70% 

70% 

70% 

70% 

Total  Cost  ($M) 

$471,820 

$617  089 

$623  438 

$551,098 

$514,946 

$493,681 

$498  656 

POM  FY17  version  of  the  PRO  model. 


As  showcased  in  this  chapter,  PROM-WED  provides  users  with  the  ablility 
to  efficiently  and  effectively  grow  space-filling  designs  that  produce  data  sets  of 
33  or  129  points  in  minutes.  This  means  that  33  or  129  data  points  as  shown  in 
Figure  89  are  produced  by  only  one  run  of  PROM-WED.  PROM-WED  not  only 
grows  data,  it  also  facilitates  basic  statistical  analysis  and  allows  for  further 
exploration  using  a  statistical  software  package  to  better  inform  decision  makers 
on  the  optimal  allocation  of  hundreds  of  millions  of  dollars  to  advertisements, 
enlistment  bonuses,  and  recommended  number  of  Navy  recruiters  in  the  field. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


Through  design  of  experiment  techniques,  PROM-WED  provides  PRO 
model  users  with  an  enhanced  analytic  tool  capable  of  producing  valuable 
insights  into  the  optimal  allocation  of  recruiting  resources.  Based  on  the  findings 
of  this  study,  each  research  question  presented  in  Chapter  I  is  answered. 
Recommendations  for  further  work  are  also  presented. 

A.  RESEARCH  QUESTION  1 

How  can  design  of  experiment  techniques  better  inform  decision  maker’s 
determination  of  the  optimal  and  robust  combination  of  recruiting  resources? 

Efficient  DOE  techniques  help  better  inform  decision  makers  on  the 
optimal  allocation  of  recruiting  resources  through  the  efficient  and  effective 
implementation  of  space-filling  designs.  Embedding  the  PRO  model  into  a  data 
farming  environment  provides  users  with  the  ability  to  execute  space-filling 
design  of  experiments.  Through  a  single  PROM-WED  excursion,  it  is  possible  to 
test  33  or  129  legacy  PRO  model  scenarios.  Each  excursion  is  able  to  test  how 
uncertainties  and  variations  in  controllable  and  uncontrollable  factors  may  affect 
the  allocation  of  recruiting  resources.  In  this  study,  Test  Case  1  and  Test  Case  2 
are  proof-of-concept  examples.  As  demonstrated  through  Test  Case  1,  the  most 
expensive  resource  is  the  number  of  recruiters  in  the  field.  However,  it  is 
apparent  that  the  total  cost  of  recruiting  is  highly  dependent  upon  the  allocation 
of  funds  to  advertising.  In  order  from  high  to  low  influence:  the  new  accession 
mission,  relative  pay,  and  unemployment  rate  drive  the  amount  of  resources 
allocated  to  advertising.  As  for  the  additional  policy  factors  included  in  the 
legalization  of  marijuana  scenario  explored  in  Test  Case  2,  there  is  evidence  to 
believe  that  increasing  the  percentage  of  high  quality  recruits  has  a  greater  effect 
on  the  total  cost  of  recruiting  than  the  decrease  in  QMA.  These  few  examples 
show  only  a  small  spectrum  of  the  vast  amount  of  information  that  PROM-WED 
can  provide.  Therefore,  by  using  DOE  techniques,  PROM-WED  is  able  to  grow 
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PRO  model  data  in  a  systematic  and  controlled  way.  By  controlling  variable 
uncertainties  and  interactions,  analysts  are  able  to  gain  insights  such  as  the  ones 
just  described.  These  insights  help  better  inform  decision  makers  on  determining 
the  optimal  and  robust  allocation  of  recruiting  resources. 

B.  RESEARCH  QUESTION  2 

How  can  efficient  design  of  experiment  techniques  be  incorporated  into  the  PRO 
model  for  future,  on-the-spot  risk,  and  sensitivity  analysis? 

The  PRO  model  is  embedded  into  a  data  farming  environment  through  the 
implementation  of  the  Microsoft  Excel  NOLH  DOE  algorithm  made  available  by 
the  SEED  Center  for  Data  Farming.  An  enhanced  GUI  allows  users  to  populate 
the  NOLH  DOE  worksheet  for  each  factor  they  would  like  to  vary.  The  NOLH 
DOE  algorithm  automatically  generates  values  for  either  33  or  129  levels  for 
each  variable.  Code  is  written  to  loop  over  each  combination  of  33  or  129 
different  scenarios.  The  result  is  a  data  set  of  33  or  129  PRO  model  runs  for 
each  PROM-WED  excursion.  PROM-WED  provides  automatically  generated 
analysis  in  Microsoft  Excel  for  on-the-spot  risk  and  sensitivity  analysis.  To  take 
advantage  of  the  space-filling  qualities  that  the  NOLH  DOE  provides,  results  from 
using  the  129-point  design  can  be  explored  using  any  available  software 
package,  like  JMP. 

C.  RESEARCH  QUESTION  3 

Can  an  enhanced  PRO  model  give  decision-makers  a  robust  solution  for  the 
optimal  allocation  of  recruiting  resources? 

An  enhanced  PRO  model  allows  analysts  to  understand  how  uncertainties 
and  fluctuations  in  controllable  and  uncontrollable  factors  affect  the  allocation  of 
recruiting  resources.  A  robust  solution  can  be  interpreted  through  two  lenses:  (1) 
resiliency,  or  (2)  gained  insight.  A  robust  solution  for  the  optimal  allocation  of 
recruiting  resources  in  terms  of  resiliency  is  one  that  is  not  overly  affected  by 
variations  in  uncontrollable  factors,  to  include  economic  uncertainties  such  as 
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unemployment  rates,  or  controllable  factors,  such  as  increasing  the  percentage 
of  high  quality  recruits.  Test  Case  1  provides  insights  to  decision  makers 
regarding  the  optimal  allocation  of  recruiting  resources  that  is  impervious  to  best 
case,  worst  case,  and  most  likely  economic  conditions.  For  example,  comparing 
the  program  of  record  and  PROM-WED’s  allocation  of  recruiting  resources  for 
Test  Case  1,  there  is  evidence  to  believe  that  the  pre-determined  recruiting 
allocation  budget  was  within  the  same  range  of  spending  as  PROM-WED’s 
solution. 

An  alternative  approach  to  interpreting  robustness  is  through  assessing 
the  value  of  information  gained  through  the  data.  PROM-WED  provides  analysts 
with  the  capability  to  data  farm  the  PRO  model.  Using  data  farming,  PROM-WED 
grows  PRO  model  data  in  an  efficient  and  space-filling  way.  Improved 
understanding  of  the  solution  space  can  range  from  basic  sensitivity  and  risk 
analysis  of  the  decision  variables  presented  in  PROM-WED’s  automatically 
generated  decision  support  capability,  to  gaining  insights  into  how  uncertainties 
in  input  factors  affect  the  optimal  allocation  of  recruiting  resources  using  a 
software  package  like  JMP.  Valuable  insights  like  these  help  analysts  better 
inform  decision-makers  on  how  factors  such  as  uncertain  unemployment  rates,  a 
proposed  policy  change,  or  constrained  resources  can  affect  the  optimal 
allocation  of  recruiting  resources. 

D.  FUTURE  WORK 

The  focus  of  this  research  was  to  enhance  the  existing  PRO  model  with 
an  efficient  design  of  experiments  capability.  PROM-WED  successfully  data 
farms  the  PRO  model’s  traditional  run  option.  Recommendations  for  further  work 
are  separated  into  three  sections.  The  first  section  addresses  additional  ways  to 
improve  PROM-WED.  The  second  section  addresses  the  opportunity  to  study 
and  improve  the  PRO  model’s  underlying  mathematical  construct.  The  last 
section  addresses  the  opportunity  to  enhance  any  Microsoft  Excel  based  model 
with  techniques  or  methods  employed  in  this  research. 
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1.  Capacity  Run  Capability  and  Additional  Design  Options 

Further  work  is  recommended  to  enhance  PROM-WED  with  the  addition 
of  the  capacity  run  option  along  with  more  design  of  experiment  choices.  While 
the  capacity  run  option  was  briefly  explored  as  a  part  of  this  research,  additional 
work  needs  to  be  done  to  ensure  that  the  data  farming  wrapper  correctly  enters 
input  values  in  the  appropriate  locations  within  the  PRO  model’s  simulation 
worksheets,  and  extracts  the  correct  output  data.  Once  the  data  farming  wrapper 
for  the  capacity  run  option  is  complete,  its  automatically  generated  decision 
support  capability  can  be  refined.  Figure  90  shows  a  graph  that  a  senior  analyst 
at  N1  requested  to  be  included  in  the  capacity  run’s  automatically  generated 
decision  support  analysis. 


Figure  90.  Example  Capacity  Run  Graph 


The  new  accession  mission  is  shown  in  red  and  the  expected  capacity 
with  a  95  percent  confidence  interval  is  shown  in  blue.  This  graph  illustrates 
where  the  Navy  has  either  budgeted  an  excess  or  deficient  amount  of  resources 
to  meet  the  recruiting  mission.  For  example,  in  FYs  2020  and  2021 ,  the  Navy  can 
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expect  to  recruit  approximately  36,500  new  recruits  each  year  when  only 
approximately  34,650  are  needed.  Since  the  NCO  missions  for  FYs  2016,  2017, 
and  2018  are  within  the  95  percent  confidence  interval,  there  is  evidence  to 
believe  that  the  pre-determined  allocation  of  recruiting  resources  will  be  sufficient 
for  those  FYs. 

Along  with  fully  integrating  the  capacity  run  option  into  PROM-WED,  work 
can  be  done  to  add  other  designs  to  PROM-WED’s  data  farming  wrapper.  This 
will  allow  analysts  to  explore  a  broader  realm  of  possibilities  to  gain  additional 
insights  about  the  complex  solution  space. 

2.  Recruiting  Cost  Function 

For  the  purpose  of  this  research,  it  was  assumed  that  the  PRO  model 
accurately  models  active  duty  enlisted  recruiting  resource  allocation.  If  this 
assumption  were  relaxed,  the  following  additional  research  is  suggested. 

Within  the  “black  box”  of  the  recruiting  cost  function,  elasticities  can  act  as 
another  variable  with  uncertainties.  Currently,  the  elasticities  are  updated 
annually  based  on  actual  data  from  the  previous  FY.  Therefore,  further  work  can 
be  done  to  include  elasticities  within  PROM-WED’s  data  farming  wrapper.  Also, 
future  work  can  be  done  to  explore  the  relevancy  of  the  recruiting  cost  function  in 
current  recruiting  practice. 

The  Navy  is  interested  in  incorporating  the  active  duty  officer,  reserve 
officer,  and  reserve  enlisted  recruiting  missions  into  the  PRO  model.  This  is  a 
unique  challenge  since  there  are  many  diverse  and  unique  communities  within 
the  active  duty  officer  corps  alone  that  require  targeted  recruiting  initiatives.  For 
example,  Navy  Doctors  are  often  incentivized  to  join  the  Navy  through  a  loan 
repayment  program  that  alleviates  medical  school  debt,  or  signing  bonuses.  On 
the  other  hand,  loan  repayment  programs  and  signing  bonuses  are  not  available 
to  prospective  general  line  officers.  Consequently,  to  recruit  general  line  officers, 
large  amounts  of  recruiting  resources  may  be  allocated  to  advertising  in  order  to 
pay  for  college  career  fair  booths. 
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Therefore,  future  work  can  be  done  to  adapt  the  recruiting  cost  function  to 
model  the  attributes  of  each  unique  recruiting  mission.  This  additional  work  will 
provide  analysts  with  an  enhanced  model  that  can  help  decision  makers 
determine  the  optimal  allocation  of  recruiting  resources  for  the  full  spectrum  of 
Navy  recruiting. 

3.  Apply  Data  Farming  to  Another  Model! 

The  methodology  used  to  develop  PROM-WED  can  be  applied  to  any 
model  built  in  Microsoft  Excel.  The  NOLH  DOE  algorithms  can  be  embedded  into 
any  Microsoft  Excel  model.  Code  similar  to  what  is  found  in  Appendix  B  can  be 
written  to  loop  over  each  design  point  of  the  NOLH.  The  resulting  product  is  an 
enhanced  tool  that  provides  an  efficient  way  to  construct,  run,  and  analyze  a 
model  using  space-filling  experimental  designs. 
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APPENDIX  A.  129-POINT  NOLH  DOE  WORKSHEET 
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APPENDIX  B.  DATA  FARMING  SUBROUTINE 


Option  Explicit 
Sub  NOLH33loop() 

‘FY  Loop 

Dim  wsNames  As  Variant 
Dim  wsCurrent  As  Variant 
Dim  I  As  Long 
Dim  j  As  Long 

wsNames  =  Array(“Sheet6,”  “Sheetl  0,”  “Sheetl  1 “Sheetl  2,”  “Sheetl  3,” 
“Sheetl  4,”  “Sheetl  5”) 

For  Each  wsCurrent  In  wsNames 

‘With  Worksheets(wsCurrent) 

If  wsCurrent  =  “Sheet6”  Then  Call  NOLH33input15 
If  wsCurrent  =  “Sheetl  0”  Then  Call  NOLH33input1 6 
If  wsCurrent  =  “Sheetl  1  ”  Then  Call  NOLFI33input1 7 
If  wsCurrent  =  “Sheetl  2”  Then  Call  NOLH33input1 8 
If  wsCurrent  =  “Sheetl  3”  Then  Call  NOLH33input1 9 
If  wsCurrent  =  “Sheetl  4”  Then  Call  NOLH33input20 
If  wsCurrent  =  “Sheetl  5”  Then  Call  NOLH33input21 
Next  wsCurrent 

End  Sub 


Sub  NOLH33input15() 

Dim  iterationNum  As  Long 

‘Update  Model  year  on  Sim  Tab 
Sheet9.Range(“B3”)  =  Sheet6.Range(“A7”) 

For  iterationNum  =  1  To  33 

‘Ed  Benefits 

Sheet5.Range(“D17”)  =  Sheet6.Range(“B”  &  4  +  iterationNum) 

‘UE  Rates 

Sheet9.Range(“C50”)  =  100  *  Sheet6.Range(“C”  &  4  +  iterationNum) 
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‘Recruiters 

Sheet5.Range(“D19”)  =  Sheet6.Range(“D”  &  4  +  iterationNum) 

‘LRP 

Sheet5.Range(“D20”)  =  Sheet6.Range(“E”  &  4  +  iterationNum) 

‘Advertising 

Sheet5.Range(“D21”)  =  Sheet6.Range(“F”  &  4  +  iterationNum) 

‘EB 

Sheet5.Range(“D22”)  =  Sheet6.Range(“G”  &  4  +  iterationNum) 

‘NCO 

Sheet9.Range(“E1 1”)  =  Sheet6.Range(“H”  &  4  +  iterationNum) 
‘Sheet5.Range(“D23”)  =  Sheet6.Range(“H”  &  4  +  iterationNum) 
‘Sheet5.Range(“D12”)  =  Sheet6.Range(“H”  &  4  +  iterationNum) 

‘TSC  l-IIIA 

Sheet5.Range(“N17”)  =  Sheet6.Range(“l”  &  4  +  iterationNum) 

‘HSDG 

Sheet5.Range(“N16”)  =  Sheet6.Range(“J”  &  4  +  iterationNum) 

‘Relative  Pay 

Sheet5.Range(“D24”)  =  Sheet6.Range(“K”  &  4  +  iterationNum) 

‘QMA 

Sheet5.Range(“D25”)  =  Sheet6.Range(“L”  &  4  +  iterationNum) 

‘  ‘Update  Model  year  on  Sim  Tab 
‘  Sheet9.Range(“B3”)  =  Sheet6.Range(“A7”) 

Call  RunTraditional6 

Sheet24.Range(“B”  &  1  +  iterationNum)  =  Sheet3.Range(“D3”)  ‘NCO  ->  output 
‘Sheet24.Range(“B”  &  1  +  iterationNum)  =  Sheet3.Range(“D4”)  ‘NCO  cap  -> 
output 

Sheet24.Range(“C”  &  1  +  iterationNum)  =  Sheet3.Range(“D5”)  ‘Unemployment  - 
>  output 

Sheet24.Range(“D”  &  1  +  iterationNum)  =  Sheet3.Range(“D6”)  ‘total  recruiters  -> 
output 

Sheet24.Range(“E”  &  1  +  iterationNum)  =  Sheet3.Range(“D7”)  ‘total  recruiters 
cost  to  output 

Sheet24.Range(“F”  &  1  +  iterationNum)  =  Sheet3.Range(“D8”)  ‘advertising  $  to 
output 
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Sheet24.Range(“G”  &  1  +  iterationNum)  =  Sheet3.Range(“D9”)  ‘EB  $  to  output 
Sheet24.Range(“H”  &  1  +  iterationNum)  =  Sheet3.Range(“D10”)  ‘ED  $  to  output 
to  output 

Sheet24.Range(T  &  1  +  iterationNum)  =  Sheet6.Range(“E”  &  4  +  iterationNum) 
‘wsResultsFY15.Range(T  &  1  +  designNumber)  =  plnputWorksheet.Cells(7,  20 
+  iSimNumber)  ‘LRP  $  -  a  constant  to  output 

Sheet24.Range(“J”  &  1  +  iterationNum)  =  Sheet3.Range(“D12”)  ‘HSDG%  to 
output 

Sheet24.Range(“K”  &  1  +  iterationNum)  =  Sheet3.Range(“D13”)  ‘UMG%  to 
output 

Sheet24.Range(“L”  &  1  +  iterationNum)  =  Sheet6.Range(“K”  &  4  +  iterationNum) 
‘Relative  Pay 

Sheet24.Range(“M”  &  1  +  iterationNum)  =  Sheet6.Range(“L”  &  4  +  iterationNum) 
‘QMA 

Sheet24.Range(“N”  &  1  +  iterationNum)  =  Sheet6.Range(“A7”) 

Next  iterationNum 
End  Sub 

Sub  NOLH33input16() 

Dim  iterationNum  As  Long 

‘Update  Model  year  on  Sim  Tab 
Sheet9.Range(“B3”)  =  Sheet10.Range(“A7”) 

For  iterationNum  =  1  To  33 

‘Ed  Benefits 

Sheet5.Range(“E17”)  =  Sheet10.Range(“B”  &  4  +  iterationNum) 

‘UE  Rates 

Sheet9.Range(“C50”)  =  100  *  Sheet10.Range(“C”  &  4  +  iterationNum) 

‘Recruiters 

Sheet5.Range(“E19”)  =  Sheet10.Range(“D”  &  4  +  iterationNum) 

‘LRP 

Sheet5.Range(“D20”)  =  Sheet10.Range(“E”  &  4  +  iterationNum) 

‘Advertising 

Sheet5.Range(“E21”)  =  Sheet10.Range(“F”  &  4  +  iterationNum) 

‘EB 


111 


Sheet5.Range(“E22”)  =  Sheet10.Range(“G”  &  4  +  iterationNum) 

‘NCO 

Sheet9.Range(“E1 1”)  =  Sheet10.Range(“H”  &  4  +  iterationNum) 
‘Sheet5.Range(“D23”)  =  Sheet10.Range(“H”  &  4  +  iterationNum) 

‘TSC  l-IIIA 

Sheet5.Range(“N17”)  =  Sheet10.Range(“l”  &  4  +  iterationNum) 

‘HSDG 

Sheet5.Range(“N16”)  =  Sheet10.Range(“J”  &  4  +  iterationNum) 

‘Relative  Pay 

Sheet5.Range(“D24”)  =  Sheet10.Range(“K”  &  4  +  iterationNum) 

‘QMA 

Sheet5.Range(“D25”)  =  Sheet10.Range(“L”  &  4  +  iterationNum) 

Call  RunTraditional6 

Sheet24.Range(“B”  &  34  +  iterationNum)  =  Sheet3.Range(“D3”)  ‘NCO  ->  output 
‘Sheet24.Range(“B”  &  34  +  iterationNum)  =  Sheet3.Range(“D4”)  ‘NCO  cap  -> 
output 

Sheet24.Range(“C”  &  34  +  iterationNum)  =  Sheet3.Range(“D5”)  ‘Unemployment 
->  output 

Sheet24.Range(“D”  &  34  +  iterationNum)  =  Sheet3.Range(“D6”)  ‘total  recruiters  - 
>  output 

Sheet24.Range(“E”  &  34  +  iterationNum)  =  Sheet3.Range(“D7”)  ‘total  recruiters 
cost  to  output 

Sheet24.Range(“F”  &  34  +  iterationNum)  =  Sheet3.Range(“D8”)  ‘advertising  $  to 
output 

Sheet24.Range(“G”  &  34  +  iterationNum)  =  Sheet3.Range(“D9”)  ‘EB  $  to  output 
Sheet24.Range(“H”  &  34  +  iterationNum)  =  Sheet3.Range(“D10”)  ‘ED  $  to  output 
to  output 

Sheet24.Range(“l”  &  34  +  iterationNum)  =  Sheet10.Range(“E”  &  4  + 
iterationNum) 

‘wsResultsFY15.Range(T  &  1  +  designNumber)  =  plnputWorksheet.Cells(7,  20 
+  iSimNumber)  ‘LRP  $  -  a  constant  to  output 

Sheet24.Range(“J”  &  34  +  iterationNum)  =  Sheet3.Range(“D12”)  ‘FISDG%  to 
output 

Sheet24.Range(“K”  &  34  +  iterationNum)  =  Sheet3.Range(“D13”)  ‘UMG%  to 
output 

Sheet24.Range(“L”  &  34  +  iterationNum)  =  Sheet10.Range(“K”  &  4  + 
iterationNum)  ‘Relative  Pay 
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Sheet24.Range(“M”  &  34  +  iterationNum)  =  Sheet10.Range(“L”  &  4  + 
iterationNum)  ‘QMA 

Sheet24.Range(“N”  &  34  +  iterationNum)  =  Sheet10.Range(“A7”) 
Next  iterationNum 
End  Sub 


Sub  NOLH33input17() 

Dim  iterationNum  As  Long 

‘Update  Model  year  on  Sim  Tab 
Sheet9.Range(“B3”)  =  Sheetl  1  .Range(“A7”) 

For  iterationNum  =  1  To  33 

‘Ed  Benefits 

Sheet5.Range(“F17”)  =  Sheetl  1  .Range(“B”  &  4  +  iterationNum) 

‘UE  Rates 

Sheet9.Range(“C50”)  =  100  *  Sheetl  1  .Range(“C”  &  4  +  iterationNum) 
‘Recruiters 

Sheet5.Range(“F1 9”)  =  Sheetl  1  .Range(“D”  &  4  +  iterationNum) 

‘LRP 

Sheet5.Range(“D20”)  =  Sheetl  1  .Range(“E”  &  4  +  iterationNum) 
‘Advertising 

Sheet5.Range(“F21”)  =  Sheetl  1  .Range(“F”  &  4  +  iterationNum) 

‘EB 

Sheet5.Range(“F22”)  =  Sheetl  1  .Range(“G”  &  4  +  iterationNum) 

‘NCO 

Sheet9.Range(“E11”)  =  Sheetl  1  .Range(“FI”  &  4  +  iterationNum) 
‘Sheet5.Range(“D23”)  =  Sheetl  1  .Range(“FI”  &  4  +  iterationNum) 

‘TSC  l-IIIA 

Sheet5.Range(“N17”)  =  Sheetl  1  .Range(“l”  &  4  +  iterationNum) 


‘HSDG 

Sheet5.Range(“N16”)  =  Sheetl  1  .Range(“J”  &  4  +  iterationNum) 
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‘Relative  Pay 

Sheet5.Range(“D24”)  =  Sheetl  1  .Range(“K”  &  4  +  iterationNum) 

‘QMA 

Sheet5.Range(“D25”)  =  Sheetl  1  .Range(“L”  &  4  +  iterationNum) 

Call  RunTraditional6 

Sheet24.Range(“B”  &  67  +  iterationNum)  =  Sheet3.Range(“D3”)  ‘NCO  ->  output 
Sheet24.Range(“C”  &  67  +  iterationNum)  =  Sheet3.Range(“D5”)  ‘Unemployment 
->  output 

Sheet24.Range(“D”  &  67  +  iterationNum)  =  Sheet3.Range(“D6”)  ‘total  recruiters  - 
>  output 

Sheet24.Range(“E”  &  67  +  iterationNum)  =  Sheet3.Range(“D7”)  ‘total  recruiters 
cost  to  output 

Sheet24.Range(“F”  &  67  +  iterationNum)  =  Sheet3.Range(“D8”)  ‘advertising  $  to 
output 

Sheet24.Range(“G”  &  67  +  iterationNum)  =  Sheet3.Range(“D9”)  ‘EB  $  to  output 
Sheet24.Range(“H”  &  67  +  iterationNum)  =  Sheet3.Range(“D10”)  ‘ED  $  to  output 
to  output 

Sheet24.Range(“l”  &  67  +  iterationNum)  =  Sheetl  1  .Range(“E”  &  4  + 
iterationNum) 

‘wsResultsFY15.Range(T  &  1  +  designNumber)  =  plnputWorksheet.Cells(7,  20 
+  iSimNumber)  ‘LRP  $  -  a  constant  to  output 

Sheet24.Range(“J”  &  67  +  iterationNum)  =  Sheet3.Range(“D12”)  ‘FISDG%  to 
output 

Sheet24.Range(“K”  &  67  +  iterationNum)  =  Sheet3.Range(“D13”)  ‘UMG%  to 
output 

Sheet24.Range(“L”  &  67  +  iterationNum)  =  Sheetl  1  .Range(“K”  &  4  + 
iterationNum)  ‘Relative  Pay 

Sheet24.Range(“M”  &  67  +  iterationNum)  =  Sheetl  1  .Range(“L”  &  4  + 
iterationNum)  ‘QMA 

Sheet24.Range(“N”  &  67  +  iterationNum)  =  Sheetl  1  .Range(“A7”) 

Next  iterationNum 
End  Sub 

Sub  NOLFI33input18() 

Dim  iterationNum  As  Long 

‘Update  Model  year  on  Sim  Tab 
Sheet9.Range(“B3”)  =  Sheetl  2.  Range(“A7”) 
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For  iterationNum  =  1  To  33 


‘Ed  Benefits 

Sheet5.Range(“G17”)  =  Sheet12.Range(“B”  &  4  +  iterationNum) 

‘UE  Rates 

Sheet9.Range(“C50”)  =  100  *  Sheet12.Range(“C”  &  4  +  iterationNum) 
‘Recruiters 

Sheet5.Range(“G19”)  =  Sheet12.Range(“D”  &  4  +  iterationNum) 

‘LRP 

Sheet5.Range(“D20”)  =  Sheet12.Range(“E”  &  4  +  iterationNum) 
‘Advertising 

Sheet5.Range(“G21”)  =  Sheet12.Range(“F”  &  4  +  iterationNum) 

‘EB 

Sheet5.Range(“G22”)  =  Sheet12.Range(“G”  &  4  +  iterationNum) 

‘NCO 

Sheet9.Range(“E1 1”)  =  Sheet12.Range(“FI”  &  4  +  iterationNum) 
‘Sheet5.Range(“D23”)  =  Sheet12.Range(“FI”  &  4  +  iterationNum) 


‘TSC  l-IIIA 

Sheet5.Range(“N17”)  =  Sheet12.Range(“l”  &  4  +  iterationNum) 

‘HSDG 

Sheet5.Range(“N16”)  =  Sheet12.Range(“J”  &  4  +  iterationNum) 

‘Relative  Pay 

Sheet5.Range(“D24”)  =  Sheet12.Range(“K”  &  4  +  iterationNum) 

‘QMA 

Sheet5.Range(“D25”)  =  Sheet12.Range(“L”  &  4  +  iterationNum) 

Call  RunTraditional6 

Sheet24.Range(“B”  &  100  +  iterationNum)  =  Sheet3.Range(“D3”)  ‘NCO  ->  output 
Sheet24.Range(“C”  &  100  +  iterationNum)  =  Sheet3.Range(“D5”) 

‘Unemployment  ->  output 

Sheet24.Range(“D”  &  100  +  iterationNum)  =  Sheet3.Range(“D6”)  ‘total  recruiters 
->  output 

Sheet24.Range(“E”  &  100  +  iterationNum)  =  Sheet3.Range(“D7”)  ‘total  recruiters 
cost  to  output 
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Sheet24.Range(“F”  &  100  +  iterationNum)  =  Sheet3.Range(“D8”)  ‘advertising  $ 
to  output 

Sheet24.Range(“G”  &  100  +  iterationNum)  =  Sheet3.Range(“D9”)  ‘EB  $  to  output 
Sheet24.Range(“H”  &  100  +  iterationNum)  =  Sheet3.Range(“D10”)  ‘ED  $  to 
output  to  output 

Sheet24.Range(“l”  &  100  +  iterationNum)  =  Sheet12.Range(“E”  &  4  + 
iterationNum) 

‘wsResultsFY15.Range(T  &  1  +  designNumber)  =  plnputWorksheet.Cells(7,  20 
+  iSimNumber)  ‘LRP  $  -  a  constant  to  output 

Sheet24.Range(“J”  &  100  +  iterationNum)  =  Sheet3.Range(“D12”)  ‘FISDG%to 
output 

Sheet24.Range(“K”  &  100  +  iterationNum)  =  Sheet3.Range(“D13”)  ‘UMG%to 
output 

Sheet24.Range(“L”  &  100  +  iterationNum)  =  Sheet12.Range(“K”  &  4  + 
iterationNum)  ‘Relative  Pay 

Sheet24.Range(“M”  &  100  +  iterationNum)  =  Sheet12.Range(“L”  &  4  + 
iterationNum)  ‘QMA 

Sheet24.Range(“N”  &  100  +  iterationNum)  =  Sheet12.Range(“A7”) 

Next  iterationNum 
End  Sub 

Sub  NOLH33input19() 

Dim  iterationNum  As  Long 

‘Update  Model  year  on  Sim  Tab 
Sheet9.Range(“B3”)  =  Sheet13.Range(“A7”) 

For  iterationNum  =  1  To  33 

‘Ed  Benefits 

Sheet5.Range(“H17”)  =  Sheet13.Range(“B”  &  4  +  iterationNum) 

‘UE  Rates 

Sheet9.Range(“C50”)  =  100  *  Sheet13.Range(“C”  &  4  +  iterationNum) 

‘Recruiters 

Sheet5.Range(“FI19”)  =  Sheet13.Range(“D”  &  4  +  iterationNum) 

‘LRP 

Sheet5.Range(“D20”)  =  Sheet13.Range(“E”  &  4  +  iterationNum) 

‘Advertising 
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Sheet5.Range(“H21”)  =  Sheet13.Range(“F”  &  4  +  iterationNum) 

‘EB 

Sheet5.Range(“H22”)  =  Sheet13.Range(“G”  &  4  +  iterationNum) 

‘NCO 

Sheet9.Range(“E1 1”)  =  Sheet13.Range(“H”  &  4  +  iterationNum) 
‘Sheet5.Range(“D23”)  =  Sheet13.Range(“H”  &  4  +  iterationNum) 

‘TSC  l-IIIA 

Sheet5.Range(“N17”)  =  Sheet13.Range(T  &  4  +  iterationNum) 

‘HSDG 

Sheet5.Range(“N16”)  =  Sheet13.Range(“J”  &  4  +  iterationNum) 

‘Relative  Pay 

Sheet5.Range(“D24”)  =  Sheet13.Range(“K”  &  4  +  iterationNum) 

‘QMA 

Sheet5.Range(“D25”)  =  Sheet13.Range(“L”  &  4  +  iterationNum) 

Call  RunTraditional6 

Sheet24.Range(“B”  &  133  +  iterationNum)  =  Sheet3.Range(“D3”)  ‘NCO  ->  output 
Sheet24.Range(“C”  &  133  +  iterationNum)  =  Sheet3.Range(“D5”) 

‘Unemployment  ->  output 

Sheet24.Range(“D”  &  133  +  iterationNum)  =  Sheet3.Range(“D6”)  ‘total  recruiters 
->  output 

Sheet24.Range(“E”  &  133  +  iterationNum)  =  Sheet3.Range(“D7”)  ‘total  recruiters 
cost  to  output 

Sheet24.Range(“F”  &  133  +  iterationNum)  =  Sheet3.Range(“D8”)  ‘advertising  $ 
to  output 

Sheet24.Range(“G”  &  133  +  iterationNum)  =  Sheet3.Range(“D9”)  ‘EB  $  to  output 
Sheet24.Range(“FI”  &  133  +  iterationNum)  =  Sheet3.Range(“D10”)  ‘ED  $  to 
output  to  output 

Sheet24.Range(“l”  &  133  +  iterationNum)  =  Sheet13.Range(“E”  &  4  + 
iterationNum) 

‘wsResultsFY15.Range(T  &  1  +  designNumber)  =  plnputWorksheet.Cells(7,  20 
+  iSimNumber)  ‘LRP  $  -  a  constant  to  output 

Sheet24.Range(“J”  &  133  +  iterationNum)  =  Sheet3.Range(“D12”)  ‘FISDG%to 
output 

Sheet24.Range(“K”  &  133  +  iterationNum)  =  Sheet3.Range(“D13”)  ‘UMG%to 
output 

Sheet24.Range(“L”  &  133  +  iterationNum)  =  Sheet13.Range(“K”  &  4  + 
iterationNum)  ‘Relative  Pay 
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Sheet24.Range(“M”  &  133  +  iterationNum)  =  Sheet13.Range(“L”  &  4  + 
iterationNum)  ‘QMA 

Sheet24.Range(“N”  &  133  +  iterationNum)  =  Sheet13.Range(“A7”) 
Next  iterationNum 
End  Sub 

Sub  NOLH33input20() 

Dim  iterationNum  As  Long 

‘Update  Model  year  on  Sim  Tab 
Sheet9.Range(“B3”)  =  Sheet14.Range(“A7”) 

For  iterationNum  =  1  To  33 

‘Ed  Benefits 

Sheet5.Range(“l17”)  =  Sheet14.Range(“B”  &  4  +  iterationNum) 

‘UE  Rates 

Sheet9.Range(“C50”)  =  100  *  Sheet14.Range(“C”  &  4  +  iterationNum) 
‘Recruiters 

Sheet5.Range(“l19”)  =  Sheet14.Range(“D”  &  4  +  iterationNum) 

‘LRP 

Sheet5.Range(“D20”)  =  Sheet14.Range(“E”  &  4  +  iterationNum) 
‘Advertising 

Sheet5.Range(“l21”)  =  Sheet14.Range(“F”  &  4  +  iterationNum) 

‘EB 

Sheet5.Range(“l22”)  =  Sheet14.Range(“G”  &  4  +  iterationNum) 

‘NCO 

Sheet9.Range(“E11”)  =  Sheet14.Range(“FI”  &  4  +  iterationNum) 
‘Sheet5.Range(“D23”)  =  Sheet14.Range(“FI”  &  4  +  iterationNum) 

‘TSC  l-IIIA 

Sheet5.Range(“N17”)  =  Sheet14.Range(“l”  &  4  +  iterationNum) 

‘HSDG 

Sheet5.Range(“N16”)  =  Sheet14.Range(“J”  &  4  +  iterationNum) 
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‘Relative  Pay 

Sheet5.Range(“D24”)  =  Sheet14.Range(“K”  &  4  +  iterationNum) 

‘QMA 

Sheet5.Range(“D25”)  =  Sheet14.Range(“L”  &  4  +  iterationNum) 

Call  RunTraditional6 

Sheet24.Range(“B”  &  166  +  iterationNum)  =  Sheet3.Range(“D3”)  ‘NCO  ->  output 
Sheet24.Range(“C”  &  166  +  iterationNum)  =  Sheet3.Range(“D5”) 

‘Unemployment  ->  output 

Sheet24.Range(“D”  &  166  +  iterationNum)  =  Sheet3.Range(“D6”)  ‘total  recruiters 
->  output 

Sheet24.Range(“E”  &  166  +  iterationNum)  =  Sheet3.Range(“D7”)  ‘total  recruiters 
cost  to  output 

Sheet24.Range(“F”  &  166  +  iterationNum)  =  Sheet3.Range(“D8”)  ‘advertising  $ 
to  output 

Sheet24.Range(“G”  &  166  +  iterationNum)  =  Sheet3.Range(“D9”)  ‘EB  $  to  output 
Sheet24.Range(“H”  &  166  +  iterationNum)  =  Sheet3.Range(“D10”)  ‘ED  $  to 
output  to  output 

Sheet24.Range(“l”  &  166  +  iterationNum)  =  Sheet14.Range(“E”  &  4  + 
iterationNum) 

‘wsResultsFY15.Range(T  &  1  +  designNumber)  =  plnputWorksheet.Cells(7,  20 
+  iSimNumber)  ‘LRP  $  -  a  constant  to  output 

Sheet24.Range(“J”  &  166  +  iterationNum)  =  Sheet3.Range(“D12”)  ‘FISDG%to 
output 

Sheet24.Range(“K”  &  166  +  iterationNum)  =  Sheet3.Range(“D13”)  ‘UMG%to 
output 

Sheet24.Range(“L”  &  166  +  iterationNum)  =  Sheet14.Range(“K”  &  4  + 
iterationNum)  ‘Relative  Pay 

Sheet24.Range(“M”  &  166  +  iterationNum)  =  Sheet14.Range(“L”  &  4  + 
iterationNum)  ‘QMA 

Sheet24.Range(“N”  &  166  +  iterationNum)  =  Sheet14.Range(“A7”) 

Next  iterationNum 
End  Sub 

Sub  NOLH33input21() 

Dim  iterationNum  As  Long 

‘Update  Model  year  on  Sim  Tab 
Sheet9.Range(“B3”)  =  Sheet15.Range(“A7”) 
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For  iterationNum  =  1  To  33 


‘Ed  Benefits 

Sheet5.Range(“J17”)  =  Sheet15.Range(“B”  &  4  +  iterationNum) 

‘UE  Rates 

Sheet9.Range(“C50”)  =  100  *  Sheet15.Range(“C”  &  4  +  iterationNum) 
‘Recruiters 

Sheet5.Range(“J19”)  =  Sheet15.Range(“D”  &  4  +  iterationNum) 

‘LRP 

Sheet5.Range(“D20”)  =  Sheet15.Range(“E”  &  4  +  iterationNum) 
‘Advertising 

Sheet5.Range(“J21”)  =  Sheet15.Range(“F”  &  4  +  iterationNum) 

‘EB 

Sheet5.Range(“J22”)  =  Sheet15.Range(“G”  &  4  +  iterationNum) 

‘NCO 

Sheet9.Range(“E1 1”)  =  Sheet15.Range(“FI”  &  4  +  iterationNum) 
‘Sheet5.Range(“D23”)  =  Sheet15.Range(“FI”  &  4  +  iterationNum) 


‘TSC  l-IIIA 

Sheet5.Range(“N17”)  =  Sheet15.Range(“l”  &  4  +  iterationNum) 

‘HSDG 

Sheet5.Range(“N16”)  =  Sheet15.Range(“J”  &  4  +  iterationNum) 

‘Relative  Pay 

Sheet5.Range(“D24”)  =  Sheet15.Range(“K”  &  4  +  iterationNum) 

‘QMA 

Sheet5.Range(“D25”)  =  Sheet15.Range(“L”  &  4  +  iterationNum) 

Call  RunTraditional6 

Sheet24.Range(“B”  &  199  +  iterationNum)  =  Sheet3.Range(“D3”)  ‘NCO  ->  output 
Sheet24.Range(“C”  &  199  +  iterationNum)  =  Sheet3.Range(“D5”) 

‘Unemployment  ->  output 

Sheet24.Range(“D”  &  199  +  iterationNum)  =  Sheet3.Range(“D6”)  ‘total  recruiters 
->  output 

Sheet24.Range(“E”  &  199  +  iterationNum)  =  Sheet3.Range(“D7”)  ‘total  recruiters 
cost  to  output 
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Sheet24.Range(“F”  &  199  +  iterationNum)  =  Sheet3.Range(“D8”)  ‘advertising  $ 
to  output 

Sheet24.Range(“G”  &  199  +  iterationNum)  =  Sheet3.Range(“D9”)  ‘EB  $  to  output 
Sheet24.Range(“H”  &  199  +  iterationNum)  =  Sheet3.Range(“D10”)  ‘ED  $  to 
output  to  output 

Sheet24.Range(T  &  199  +  iterationNum)  =  Sheet15.Range(“E”  &  4  + 
iterationNum) 

‘wsResultsFY15.Range(T  &  1  +  designNumber)  =  plnputWorksheet.Cells(7,  20 
+  iSimNumber)  ‘LRP  $  -  a  constant  to  output 

Sheet24.Range(“J”  &  199  +  iterationNum)  =  Sheet3.Range(“D12”)  ‘HSDG%to 
output 

Sheet24.Range(“K”  &  199  +  iterationNum)  =  Sheet3.Range(“D13”)  ‘UMG%to 
output 

Sheet24.Range(“L”  &  199  +  iterationNum)  =  Sheet15.Range(“K”  &  4  + 
iterationNum)  ‘Relative  Pay 

Sheet24.Range(“M”  &  199  +  iterationNum)  =  Sheet15.Range(“L”  &  4  + 
iterationNum)  ‘QMA 

Sheet24.Range(“N”  &  199  +  iterationNum)  =  Sheet15.Range(“A7”) 

Next  iterationNum 
End  Sub 

Sub  RunTraditional6() 

Dim  pCalcWorksheet  As  Worksheet 

Dim  pResultWorksheet  As  Worksheet 

Dim  plnputWorksheet  As  Worksheet 

Dim  pllserlnterfaceWorksheet  As  Worksheet 

Dim  iSimNumber  As  Long 

Dim  iNumSimulations  As  Long 

Dim  iOldCalcalculationSetting  As  Long 

Dim  pUserWorksheet  As  Worksheet 

Dim  pTradRunsWorksheet  As  Worksheet 

Dim  casenum  As  Long 

Dim  designPoints  As  Long 

Set  pCalcWorksheet  =  ThisWorkbook.Worksheets(“Simulation”) 

Set  pResultWorksheet  =  ThisWorkbook.Worksheets(“Output”) 

Set  plnputWorksheet  =  ThisWorkbook.Worksheets(“lnput”) 

Set  pUserlnterfaceWorksheet  =  ThisWorkbook.Worksheets(“User  Interface”) 

Set  pTradRunsWorksheet  =  ThisWorkbook.Worksheets(“Traditional  Runs”) 
iOldCalcalculationSetting  =  Application. Calculation 
iNumSimulations  =  7 

Application. ScreenUpdating  =  False 
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Application. Calculation  =  xlCalculationManual 
pResultWorksheet.Columns(“B:Q”). Clear 
ThisWorkbook.Worksheets(“Simulation”).  Activate 
pCalcWorksheet.Cells(8,  2)  =  “User  Defined” 

‘pCalcWorksheet.Cells(8,  2)  =  “Model  Year” 
pResultWorksheet.Ce!is(2,  3)  =  “Resource  Run” 
pCalcWorksheet.Cells(14,  2)  =  pUserlnterfaceWorksheet.Cells(27,  4) 
pCalcWorksheet.Cells(15,  2)  =  pUserlnterfaceWorksheet.Cells(28,  4) 
pCalcWorksheet.Cells(16,  2)  =  pUserlnterfaceWorksheet.Cells(29,  4) 
pCalcWorksheet.Cells(17,  2)  =  pUserlnterfaceWorksheet.Cells(30,  4) 

For  casenum  =  1  To  3  ‘Run  through  High  UE,  Base  UE,  Low  UE  scenarios 
ThisWorkbook.Worksheets(“User  Interface”). Activate 
pUserlnterfaceWorksheet.Cells(18,  17)  =  casenum 
ThisWorkbook.Worksheets(“Simulation”).  Activate 


‘Just  for  FY  2015 
For  iSimNumber  =  1  To  1 

‘pCalcWorksheet.Cells(3,  2)  =  pCalcWorksheet.Cells(9  +  iSimNumber,  29) 

‘Updates  the  model  year 

Application. Calculate  ‘Recalculates  sheet 

‘pResultWorksheet.Cells(2,  3  +  iSimNumber)  =  iSimNumber  +  2014  ‘Copies 
Model  Year  to  output 

pResultWorksheet.Cells(3,  3  +  iSimNumber)  =  pCalcWorksheet.Cells(1 1 , 5)  ‘4 
‘Copies  NCO  to  output 

pResultWorksheet.Cells(4,  3  +  iSimNumber)  =  “N/A”  ‘Copies  Capacity  to  output 
pResultWorksheet.Cells(5,  3  +  iSimNumber)  =  pCalcWorksheet.Cells(50,  3) 
‘Copies  unemployment  to  output 

pResuitWorksheet.Cells(6,  3  +  iSimNumber)  =  pCalcWorksheet.Cells(8,  9) 
‘Copies  total  recruiters  to  output 

pResultWorksheet.Cells(7,  3  +  iSimNumber)  =  pCalcWorksheet.Cells(8,  10) 
‘Copies  total  recruiters  cost  to  output 

pResultWorksheet.Cells(8,  3  +  iSimNumber)  =  pCalcWorksheet.Cells(9,  10) 
‘Copies  advertising  $  to  output 

pResultWorksheet.Cells(9,  3  +  iSimNumber)  =  pCalcWorksheet.Cells(10,  10) 
‘Copies  EB  $  to  output 

pResultWorksheet.Cells(10,  3  +  iSimNumber)  =  pCalcWorksheet.Cells(1 1 ,  10) 
‘Copies  ED  $  to  output  to  output 

pResultWorksheet.Cells(1 1,3  +  iSimNumber)  =  plnputWorksheet.Cells(7,  20  + 
iSimNumber)  ‘Copies  LRP  $  -  a  constant  to  output 

pResultWorksheet.Cells(12,  3  +  iSimNumber)  =  pCalcWorksheet.Cells(9,  6) 
‘Copies  HSDG%  to  output 
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pResultWorksheet.Cells(13,  3  +  iSimNumber)  =  pCalcWorksheet.Cells(1 1 , 6) 
‘Copies  UMG%  to  output 
‘pResultWorksheet.Cells(14,  3  +  iSimNumber)  = 
WorksheetFunction.Sum(pResultWorksheet.Cells(7,  3  +  iSimNumber), 
pResultWorksheet.Cells(8,  3  +  iSimNumber),  pResultWorksheet.Cells(9,  3  + 
iSimNumber),  pResuitWorksheet.Cells(10,  3  +  iSimNumber), 
pResultWorksheet.Cells(1 1 , 3  +  iSimNumber)) 

ThisWorkbook. Worksheets(“T raditional  Runs”). Activate 
pTradRunsWorksheet.Ce!ls(7  +  casenum,  1  +  iSimNumber)  = 
pResultWorksheet.Cel!s(8,  3  +  iSimNumber)  ‘Also  enter  Capacity  in  UE 
scenarios  table 

ThisWorkbook.  Worksheets(“Simulation”).  Activate 

Next 

Next 

ThisWorkbook.Sheets(“Output”).  Activate 
Polished  ‘Formats  output 
‘ResourceChart 

Application. Calculation  =  iOldCalcalculationSetting 
Application. StatusBar  =  False 
Application. Screenllpdating  =  True 
ThisWorkbook.Sheets(“Output”).  Activate 
ActiveSheet.Cells(1 ,  1). Select 

End  Sub 
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APPENDIX  C.  PROM-WED  USER  MANUAL 


Planned  Resource  Optimization  Model 
with  Experimental  Design 

(PROM-WED) 
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NOTICE: 

The  user  is  cautioned  that  PROM-WED  has  not  undergone  formal  verification 
and  validation  testing,  and  comes  without  any  warranty.  Informal  testing  confirms 
the  outputs  from  PROM-WED  match  the  output  from  the  legacy  PRO  model. 
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I. 


WHAT  IS  PROM-WED 


PROM-WED  embeds  the  legacy  PRO  model  within  a  data  farming 
environment.  The  foundation  of  PROM-WED’s  data  farming  wrapper  is  the  nearly 
orthogonal  Latin  hypercube  (NOLH).  The  NOLH  design  of  experiments  (DOE) 
builds  experimental  designs  that  efficiently  and  effectively  explore  the  solution 
space  (Cioppa  &  Lucas,  2007).  This  good  space-filling  capability  means  that 
uncertainties  and  fluctuations  in  input  variables  along  with  multivariable 
interactions  can  be  adequately  investigated  (Sanchez  &  Wan,  2015). 

The  33  and  129  design  point  NOLH  designs  were  used  to  construct 
PROM-WED’s  data  farming  wrapper.  The  33-point  NOLH  DOE  tests  each 
variable  at  33  levels  and  grows  data  for  33  legacy  PRO  model  runs,  whereas  the 
129-point  NOLH  DOE  tests  each  variable  at  129  levels  and  grows  data  for  129 
legacy  PRO  model  runs.  PROM-WED’s  graphical  user  interface  (GUI)  allows 
users  to  easily  input  a  range  of  values  for  each  input  variable  into  the  NOLH  DOE 
worksheet,  without  need  for  knowledge  or  familiarity  with  data  farming  or  DOE 
techniques  (Sanchez,  2011). 

A  completed  PROM-WED  excursion  grows  a  data  set  for  either  33  or  129 
data  points.  Automatically  generated  sensitivity  analysis  provides  users  with  a 
basic  risk  assessment  picture  focused  on  the  decision  variables  using  the  data 
grown  by  PROM-WED.  Further  insights  into  variable  interactions  and  effects  of 
input  variables  can  be  easily  explored  using  available  data  analysis  software. 
PROM-WED  transforms  the  legacy  PRO  model  into  a  resource  that  N1  can  use 
to  gain  robust  insights  into  the  optimal  allocation  of  recruiting  resources. 
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II.  OUTPUT  OPTIONS 


PROM-WED  provides  users  with  decision  support  capabilities  to  analyze 
the  data  grown  by  each  excursion.  PROM-WED  offers  two  decision  support 
capabilities:  (A)  automatically  generated  analysis,  and  (B)  data  generated  for 
further  analysis  requiring  a  statistical  software  package. 

A.  AUTOMATICALLY  GENERATED  ANALYSIS 

PROM-WED’s  “Decision  Support  Analysis”  for  the  traditional  run  option 
provides  users  with  a  broad  understanding  of  how  variability  in  decision 
variables,  controllable  policy  changes,  and  uncontrollable  market  factors  affect 
the  total  cost  of  recruiting.  This  type  of  analysis  would  be  appropriate  for  testing 
excursions  during  a  time  constrained  meeting,  working  group,  or  whenever  basic 
analysis  needs  to  be  generated  quickly. 

An  example  of  PROM-WED’s  automatically  generated  analysis  follows. 


129 


PROM-WED 

Decision  Support  Analysis 


Traditional  Run:  33  Design  Points 


Mean  Total  Cost  of  Recruiting  over  FYDP 

600 

500  _ _ _ _ _ 

400  * _ \  ‘ _ ' _ _ _ _ _ _ _ 

300 

200 

100 

0 

2015  2016  2017  2018  2019  2020  2021 


-  -  -  Confidence 
High 

•  M  ean 

- Confidence 

Low 


OX) 

c 


3 

i- 

u 


a 

oc 


o 

w 


500 
450 
400 
^  350 

^  300 
O 

:=  250 
|  200 
^  150 
100 
50 
O 


PROMWED  Excursion  versus  Program  of  Record 


Ci 

LU 


o 

CL 

CL. 


■  Advertising 
EB 

■  Recruiter 
Cost 


a:  Ci  a: 

O  lu  O 

a.  £  a. 

O 

oc 

Q_ 


Ci 

LU 


o 

ai 

Q_ 


CL. 

O 


Cl. 


Cl 

LU 


o 

CL 

a_ 


C L 
O 


CL 


2015 


2016  2017 


201S  2019 


2020  2021 


Scenario 


2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCO 

high 

40000 

40000 

40000 

40000 

40000 

40000 

40000 

low 

30000 

30000 

30000 

30000 

30000 

30000 

30000 

LRP 

high 

7.44 

11.22 

11.28 

11.38 

TTT3 

ms 

11.67 

low 

7.44 

11.22 

11.28 

11.38 

11.43 

11.46 

11.67 

HSDG 

high 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

low 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

TSC 

hiqh 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

low 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

UE 

hiqh 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

low 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

Rei  Pay 

hi  qh 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

low 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

QUA 

hiqh 

1 883304 

1873304 

1863304 

1853304 

1843304 

1833304 

1823304 

low 

1 873304 

1863304 

1853304 

1843304 

1833304 

1823304 

1813304 

130 


PRDM-WED 

Decision  Support  Analysis 
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B.  JMP  ANALYSIS 


Analysts  will  need  to  use  a  statistical  software  package  to  take  full 
advantage  of  the  data  grown  by  PROM-WED.  Therefore,  data  produced  by 
PROM-WED  is  designed  to  be  easily  uploaded  into  a  software  package,  such  as 
JMP  (JMP  Pro,  2015). 

The  following  are  examples  of  insights  gained  through  analysis  of  PROM- 
WED  data  in  JMP. 

1.  Partition  Tree 


▼  Partition  for  Total  Cost  of  Recruiting 
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Over  80  percent  of  variance  in  the  total  cost  of  recruiting  is  explained  by  the 
amount  of  funds  allocated  to  advertising. 
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2.  Stepwise  Regression 

The  total  cost  of  recruiting  can  be  formulated  into  a  stepwise  regression  model: 


Prediction  Expression 

150.980052850998 
+  0.02553596931337  *  NCO 
+  -24.593679044673*  Unemployment 
+  -550.34801858165*  Relative  Pay 

+  ( NCO  -  35000.0620155039 )  *  ( ( Relative  Pay  - 1 )  *  -0.1270005027034  j 
+  ( NCO  -  35000.0620155039 )  *  ( ( NCO  -  35000.0620155039 )  *  0.00000304992539 1 
+  ( Relative  Pay  - 1 )  *  ( ( Relative  Pay  - 1 )  *  1882.37625427676  ] 


3.  Contour  Plots 
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The  contour  plot  indicates  that  nearly  half  of  the  solution  space  supports  a 

low  advertising  budget,  represented  by  the  dark  blue  region.  The  cost  of 

advertising  substantially  increases  when  relative  pay  favors  the  civilian  sector 

and  the  accession  mission  is  high,  represented  by  the  red  region.  Once  relative 

pay  exceeds  approximately  1 .00,  changes  in  the  new  accession  mission  have 

little  to  no  effect  on  the  amount  of  resources  allocated  to  advertising. 
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STEP-BY-STEP  INSTRUCTIONS  TO  RUN  PROM-WED 


Step  1:  Unzip  the  “PROM-WED.zip”  file,  and  save  the  “PROM-WED. xlsm”  file 
and  “NOLH.xls”  file  in  the  same  folder.  This  folder  is  where  the  output  file 
generated  by  PROM-WED  will  be  saved  following  the  PROM-WED  excursion. 


►  PROM-WED  ►  PROM-WED 
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yf  Favorites 

Name 

Date  modified 

Type  Size 

\lp  Recently  Changed 

Public 

K  Desktop 

IJ  Downloads 

Recent  Places 

I^NOLH 
fo  PROM-WED_v2 

5/20/2016  2:16  PM 

1/20/2017 11:55  AM 

Microsoft  Excel  97...  1,136  KB 

Microsoft  Excel  M...  2,157  KB 

3  Libraries 

3  Documents 
g  Pictures 

H  Videos 

Computer 

S.OHC) 

^  arhogartS  (\\comfoi 
n  My  Web  Sites  on  MS 

^  Network 

Step  2:  Open  the  PROM-WED  file,  and  ensure  the  “Enable  Content”  button  is 
selected. 


!  VMtmhr  Wwmng  *di*«  ronifwi  »«»«  d*ubir«J  Out  *»»  W*  Otto*  tnatife  Corrtr^ 

fcjpsil  Planned  Resource  Optimization  Model 

r^— ' with  Experimental  Design 

(PRQM-WED) 


Clicti  Mon  to  Lsuncti  PROM-WED 


POC: It  Altioon  Hogorth 


134 


The  first  time  you  open  PROM-WED,  the  NOLH.xls  file  link  needs  to  be  updated. 
To  do  this,  select  the  “Edit  Links...”  button. 


To  update  the  NOLH.xls  file,  click  on  the  “Change  Source...”  button. 


A  file  search  window  will  pop-up.  Navigate  to  the  folder  where  you  saved  the  files 
after  unzipping  them.  Select  the  “NOLH.xls”  file,  and  click  on  the  “OK”  button. 

v  i  >  PSOM-WED  Score*  MOM- AK>  P 
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The  “Edit  Links”  window  will  pop-up.  Once  the  “NOLH.xls”  worksheet’s  status 
updates  to  “OK,”  click  on  the  “Close”  button. 


Edit  Units 
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Step  3:  Open  the  PROM-WED  file,  and  select  the  “PROM-WED”  button  to  open 
the  GUI. 
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Step  4:  Select  the  appropriate  starting  fiscal  year  (FY)  from  the  drop-down  list. 
The  current  version  of  the  legacy  PRO  model  is  set  at  a  FY  2015  start. 


Planned  Resource  Optimization  Model 

with  Experimental  Design 

(PROM-WED) 


M. 


I  *T»17 
[  HCJ  •Coteoe  **  30  If 

J] 

I  ** 

Hsrketf+ct* r* 


<3MA 

Tjct-nu 

XDG 


j  V+rtcii  ft+rkri  factor* 


Im  Mgte 


Meet  Aim  Type 


rife* 


i — r 


137 


Step  5:  To  constrain  a  decision  variable,  select  it  from  the  list,  and  click  “Fix  DV; 
button. 
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The  default  data  from  the  legacy  PRO  Model  will  automatically  populate  the 
“Design  of  Experiments  Table.” 


Planned  Resource  Optimization  Model 

with  Experimental  Design 

(PROM  WED) 


Marl  M  TV  \  »  ^ 

ytVwtrtirt 


factor* 


*vZm  S«lRjn»e 


[.Vmjh  of  I  xpmmmlt 


S3  Daagr  tota 


r  Wn)t  «uM  *or  r+rm  r  K 


’  SMStcmr*  Harm-  Vmine 


139 


Step  6:  Input  the  range  of  values  for  the  decision  variable  in  the  “Design  of 
Experiments  Table.”  Input  the  low  value  of  the  range  in  the  “Low  Level”  text  box 
for  each  FY,  and  the  high  value  of  the  range  in  the  “High  Level”  text  box  for  each 
FY.  In  this  example,  the  number  of  recruiters  is  tested  from  2,500  to  3,500  for 
each  FY. 

Each  year  can  be  tested  using  different  ranges.  For  example,  to  represent  a 
smaller  recruiter  force  in  FY  2021 ,  the  range  could  be  inputted  as  2,000  to  2,700. 

If  you  want  to  constrain  the  decision  variable  at  the  default  value  populated  by 
the  legacy  PRO  model,  select  the  “Fix  Value”  button.  By  selecting  “Fix  Value,” 
the  default  values  for  the  decision  variable  in  the  “Design  of  Experiments  Table” 
are  deposited  into  the  NOLH  worksheet  for  each  FY.  This  decision  variable  is 
now  moved  to  the  “Fixed  Decision  Variables”  list,  and  the  “Design  of  Experiments 
Table”  is  cleared.  (If  this  is  your  course  of  action,  continue  to  Step  8.) 

If  you  want  to  constrain  the  decision  variable  at  one  number  that  is  different  than 
the  default  value  populated  by  the  legacy  PRO  model,  the  same  number  has  to 
be  inputted  into  the  “Low  Level”  and  “High  Level”  text  boxes.  For  example,  if  you 
want  to  constrain  the  number  of  recruiters  in  FY  2021  to  2700,  then  you  would 
enter  2700  in  both  the  “Low  Level,”  and  “High  Level”  text  boxes. 
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Step  7:  Once  the  “Design  of  Experiments  Table”  is  fully  populated  with  the  low 
and  high  levels  for  each  FY,  select  the  decision  variable  from  the  “Input  Values” 
box,  and  click  on  the  “Set  Range”  button. 
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By  selecting  “Set  Range,”  the  low  and  high  values  entered  for  this  decision 
variable  in  the  “Design  of  Experiments  Table”  are  deposited  into  the  NOLH 
worksheet  for  each  FY.  This  decision  variable  is  now  moved  to  the  “Fixed 
Decision  Variables”  list,  and  the  “Design  of  Experiments  Table”  is  cleared. 


fKlPCl 
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Step  8:  Follow  Steps  5-7  to  fix  any  other  decision  variables. 

Reminders: 

•  If  you  constrain  a  decision  variable  to  a  number  other  than  the  default 
values  populated  from  the  legacy  PRO  model,  as  mentioned  earlier  enter 
the  same  number  into  the  low  and  high  level  text  boxes,  and  select  the 
“Set  Range”  button  when  complete. 

•  Since  the  PRO  model  solves  an  optimization  problem,  ensure  that  at  least 
one  of  the  following  decision  variables:  Recruiters,  Advertising  or 
Enlistment  Bonus  (EB)  remain  in  a  “float”  status.  In  this  example,  only  the 
number  of  recruiters  are  fixed. 
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Step  9:  Once  all  decision  variables  that  need  to  be  fixed  are  fixed,  gears  shift  to 
the  market  factors.  The  “Market  Factors”  list  includes  all  market  factors  (relative 
pay,  QMA  and  unemployment  rate)  and  policy  factors  (percentage  of  high  quality 
recruits  (TSC  l-IIIA),  percentage  of  recruits  with  a  high  school  diploma  (HSDG), 
and  NCO).  Each  market  factor,  from  relative  pay  to  NCO,  must  either  be  fixed  at 
one  value,  or  a  range  of  values  needs  to  be  entered. 

Similar  to  how  decision  variables  are  fixed,  select  “Relative  Pay”  from  the  list  of 
market  factors,  and  select  the  “Add  MF”  button. 
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The  default  data  from  the  legacy  PRO  Model  automatically  populates  in  the 
“Design  of  Experiments  Table.” 
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Step  10:  Input  the  range  values  for  the  market  factor  in  the  “Design  of 
Experiments  Table.”  Input  the  low  value  of  the  range  in  the  “Low  Level”  text  box 
for  each  FY,  and  the  high  value  of  the  range  in  the  “High  Level”  text  box  for  each 
FY. 

In  this  example,  the  relative  pay  is  tested  from  0.8  to  1 .2  for  each  FY.  Clicking  the 
“Set  Range”  button  deposits  the  low  and  high  values  entered  for  this  market 
factor  into  the  NOLH  worksheet  for  each  FY. 
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This  market  factor  is  now  moved  to  the  “Varied  Market  Factors”  list,  and  the 
“Design  of  Experiments  Table”  is  cleared. 
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Step  11:  Work  through  each  “Market  Factor”  in  the  list,  from  “Relative  Pay”  to 
“NCO”  following  Steps  9-10. 

Note  that  each  year  can  be  tested  using  a  different  range  of  values  for  the  market 
factors.  For  example,  an  annual  decrease  of  10,000  QMA  can  be  entered  as 
shown  in  the  figure  below. 
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If  you  want  to  constrain  the  market  factor  at  one  number  different  than  what  is 
populated  by  the  legacy  PRO  model,  the  same  number  has  to  be  inputted  into 
the  “Low  Level”  and  “High  Level”  text  boxes.  Then  select  the  “Set  Range”  button. 

To  constrain  the  market  factor  at  the  value  automatically  populated  in  the  “Design 
of  Experiments  Table,”  select  the  market  factor  from  the  “Input  Values”  box,  and 
click  on  the  “Fix  Value”  button. 
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Step  12:  Work  through  all  seven  market  factors  until  they  are  all  accounted  for.  A 
market  factor  is  accounted  for  once  it  appears  in  either  the  “Varied  Market 
Factors,”  or  “Fixed  Market  Factors”  lists. 
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Step  13:  Select  “Traditional  Run”  under  “Select  Run  Type.”  (Currently,  only  the 
Traditional  Run  option  is  operational). 

Automatically  Generated  Decision  Support: 

The  “33  Design  Points”  option  is  well  suited  for  the  automatically  generated 
decision  support  analysis.  The  “129  Design  Points”  option  can  also  be  used,  but 
it  will  take  additional  time  to  run  (approximately  10  minutes  versus  2-3  minutes). 
The  “129  Design  Points”  option  grows  more  data,  resulting  in  a  narrower  95% 
confidence  interval. 

Analysis  in  JMP: 

The  “129  Design  Points”  option  is  intended  to  be  used  for  further  analysis  in  a 
commercial  statistical  software  package,  such  as  JMP. 


Planned  Resource  Optimization  Model 

with  Experimental  Design 

(PROM-WED) 


Start  in  FY  |  FY  2015 


Saved  Scenarios: 


3 


Set  Variables 
Decision  Variables 


Fix  DV 


EB 


Advertising  (AC  E ri.  Only) 


Market  Factors 


TSCI-mA 

HSOG 

Unemployment  Rate 
LRP 


u 


Input  Values 


Saba  to  Tm  t 
Smpie  Vatu* 


Saha  as  ran 
ti- c*  l  J.« 


Fix  Value 


Set  Range 


Remove 


Design  of  Experiments  Table 
Base  Low  High  Decimal 

FY  Value  Level  Level  Places 

I- 
m*  I- 

rr.7  |— 

m.  |  |  |  | 

m*  |  [— 

FV20  |  |  |  |— 

|  |— 


Variable  Set 
Fixed  Decision  Variables 

Recruters 


Varied  Market  Factors 

Fixed  Market  Factors 

Relative  Pay 

HSOG 

QMA 

LRP 

T5C I -III A 

Unemployment  Rate 

NCO(50%BoYDe>) 

Analysis  Options 

Select  Run  Type  Design  of  Experiments 

Capacity  Run 

1 

129  Depgn  Points 

r  Indude  output  for  analysis  n  JMP 


V  Save  Scenano 


Name  Scenano 


r 


Run 

NOLHRun 


Saiaa  »  rjn  spsoa&np, 
OOF  Hasten 


150 


Step  14:  To  save  PROM-WED  output  to  a  separate  .xls  file  for  analysis  in  JMP, 
select  the  “Include  output  for  analysis  in  JMP”  box.  This  will  save  the  PROM- 
WED  output  as  a  .xls  file  in  the  same  folder  that  the  PROM-WED  model  was 
saved  in. 
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Step  15:  Once  the  run  options  are  set,  select  the  “NOLH  Run”  button.  A 
message  will  pop-up  providing  an  estimated  wait  time  for  the  PROM-WED 
excursion.  Click  “OK.” 


Step  16:  When  the  PROM-WED  excursion  is  complete,  the  automatically 
generated  decision  support  analysis  will  appear  (this  is  true  for  both  the  33  and 
129  point  designs).  If  you  selected  the  option  to  output  PROM-WED  data  for 
analysis  in  JMP,  the  .xls  file  named  “PROMWED_Output129.xls”  will  appear  in 
the  folder  that  your  PROM-WED  model  is  saved  in. 


Please  be  aware  that  each  129  design  point  output  file  will  be  named 
“PROMWED_Output129.xls.”  It  is  recommended  that  you  rename  the  file  before 
running  another  PROM-WED  excursion. 
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IV.  GUIDELINES  FOR  ANALYSIS  OF  PROM-WED  DATA  IN  JMP 


Using  JMP  Pro  12,  the  following  section  provides  a  tutorial  on  analysis 
techniques  for  PROM-WED  output.  Steps  1-5  explain  how  to  upload  and  prepare 
the  data  for  analysis  in  JMP,  followed  by  guidance  on  how  to  conduct  various 
analysis  techniques. 

Analysis  Techniques: 

A.  Oneway  Analysis  of  Total  Cost  of  Recruiting  by  FY 

B.  Explore  Outliers  from  the  Oneway  Analysis  Graph 

C.  Select  one  FY  to  Analyze 

D.  Distribution 

E.  Partition  Trees 

F.  Stepwise  Regression  Model 

G.  Scatterplot  Matrix 

H.  Contour  Plot 

Step  1 :  To  load  the  PROM-WED  data  into  JMP,  select  the  folder  icon. 


File  Tables  DOE  Analyze 


c53  £  Ji  □  &  □  Jf 


Recent  Help 

Discovering  JMP 

Beginners  Tutorial 

Using  JMP 
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Step  2:  Select  the  output  data  of  interest,  select  the  “Best  Guess”  option,  and 
click  “Open.” 


Step  3:  Select  the  “Import”  button. 
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The  PROM-WED  output  data  should  appear  in  a  table,  as  shown  below: 
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#  BB  H  B-  IT*  * 

»  JMP 129DesPts 

1  B 

Run  # 

FY 

Total  Cost  of 
Recruiting 

Advertising 

EB 

Education 

Incentive 

Tol 

1 

1 

2015 

327.1265 

35.7344 

47.1694 

0 

- 

2 

2 

2015 

377.1404 

55.9153 

73.8082 

0 

M  Columns  (15/0) 

3 

3 

2015 

410.8494 

85.6254 

113.0255 

0 

A  Run  # 

4 

4 

2015 

283.2184 

20.127 

26.5676 

0 

A  FY 

5 

5 

2015 

359.4841 

43.8919 

57.9373 

0 

A  Total  Cost  of  Recruiting 

A  Advertising 

A  EB 

6 

6 

2015 

467.9692 

85.7103 

113.1376 

0 

7 

7 

2015 

316.5387 

18.2144 

24.043 

0 

8 

8 

2015 

464.2046 

77.1682 

101.862 

0 

A  Total  Recruiters 

9 

9 

2015 

404.0421 

73.583 

97.1295 

0 

A  Recruiter  Cost 

10 

10 

2015 

471.4698 

104.0235 

137.311 

0 

A  NCO 

A  LRP 

11 

11 

2015 

320.903 

30.0158 

39.6208 

0 

12 

12 

2015 

401.3847 

72.72 

95.9904 

0 

A  TSC I-IUA 

A  Unemployment 

13 

13 

2015 

316.4949 

11.3114 

14.9311 

0 

14 

14 

2015 

566.1871 

120.0318 

158.4419 

0 

A  Relative  Pay 

15 

15 

2015 

310.9625 

8.0795 

10.6649 

0 

A  QMA 

16 

16 

2015 

489.4982 

84.7874 

111.9194 

0 

©  Rows 

17 

17 

2015 

450.3778 

95.4969 

126.056 

0 

All  rows  903 

18 

18 

2015 

422.3013 

81.1709 

107.1456 

0 

Selected  0 

19 

19 

2015 

313.7766 

33.8635 

44.6998 

0 

Excluded  0 

20 

20 

2015 

332.7022 

38.4201 

50.7146 

0 

Labelled  0 

21 

21 

2015 

382.5251 

44.468 

58.6978 

0 

, 

*  1  m  ► 

Step  4:  Change  the  FY  column  from  “continuous”  to  “nominal”  data,  by  right 
clicking  on  the  blue  triangle  next  to  “FY,”  and  select  “nominal”  from  the  drop 
down  menu. 


•  JMP l290esPts 

:> 

w 

Run  * 

El 

-  Source 

1 

1 

2015 

2015 

2 

2 

•  Columns  (15/1) 

3 

3 

2015 

- 

4 

4 

2015 

*4-2 _ 

5 

5 

2015 

•  Continuous 

6 

6 

2015 

7 

7 

2015 

8 

8 

2015 

Nominal 

9 

9 

2015 

A  Recruiter  Cost 

10 

10 

2015 

A  NCO 

11 

11 

2015 

12 

12 

2015 

13 

13 

2015 

A  Unemployment 

14 

14 

2015 

A  Relativ  e  Pay 

15 

15 

2015 

A  QMA 

16 

16 

2015 

|  »  Rows 

17 

17 

2015 

All  rows  903 

18 

18 

2015 

Selected 

0 

19 

19 

2015 

Excluded 

0 

20 

20 

2015 

Hidden 

- 

21 

21 

2015 
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The  blue  triangle  next  to  FY  will  change  to  a  red  bar  chart  icon  when  JMP 
changes  its  classification  to  nominal  data. 


t»l  Columns  (15/1) 

J  Run  r _ 

ii,jS 

A  Total  Cost  of  Recruiting 
A  Advertising 
A  EB 

A  Education  Incentive 
A  Total  Recruiters 
A  Recruiter  Cost 
A  NCO 
A  LRP 
A  HSDG 
A  TSC  I-mA 
A  Unemployment 
A  Relative  Pay 
A  QMA _ 


The  data  is  now  ready  to  be  analyzed. 
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A 


ONEWAY  ANALYSIS  OF  TOTAL  COST  OF  RECRUITING  BY  FY 


»  Oneway  Analysis  of  Total  Cost  of  Recruiting  By  FY 

1000 
900 

a.  800 

_c 

I  700 

<U 

ec 

“  600 
o 
u 

1  500 

I— 
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A  Quantiles 


Level 

Minimum 

10% 

25% 

Median 

75% 

90% 

Maximum 

2015 

251.8643 

283.3929 

309.1051 

349.9401 

421.2093 

521.9235 

959.1079 

2016 

258.138 

289.8711 

315.9455 

356.89 

428,4688 

529.2777 

968.0586 

2017 

261.6095 

293.5959 

320.1883 

361.2135 

433.1215 

533.85 

974.1696 

2018 

265.1709 

297.4278 

324.5511 

365861 

437.8086 

538.5172 

980.3789 

2019 

268.7332 

301.3375 

329.0443 

370.5412 

442.3917 

543.1901 

986.5974 

2020 

272.8857 

305.6795 

334.2964 

375.5285 

447.5582 

548.4681 

993.3847 

2021 

277.2888 

310.3785 

339.8189 

380.7692 

453.0851 

554.002 2 

100043 

2015 


2016 


2017 


2018 

FY 


2019 


2020 


2021 


Step  1 :  To  create  an  oneway  analysis  of  total  cost  of  recruiting  by  FY  graph, 
select  “Analyze”  from  the  ribbon,  and  select  “Fit  Y  by  X.” 


Analyze  Graph  Tools  View  Window  Help 


I  & 

Distribution 

ILt 

Fit  Y  by  X 

r 

Matched  Pairs 

in 

Tabulate 

3m 

Fit  Model 

Modeling 

► 

Multivariate  Methods 

► 

Quality  and  Process 

¥ 

Reliability  and  Survival 

¥ 

11 

Consumer  Research 

,  ril'.l 

¥ 

;  Examine  relationships  between  two 
j  variables.  Creates  a  Oneway,  Bivariate, 
Contingency,  or  Logistic  analysis 
'  based  on  the  context  and  modeling 
type. 

u cat  ion 

icentive 

0 

0 

Tol 

83.2184 

20.127 

26.5676 

0 

59.4S41 

43.8919 

57.9373 

0 

67.9692 

85.7103 

113.1376 

0 

16.5387 

18.2144 

24.043 

0 

64.2046 

77.1682 

101.862 

0 

04.0421 

73.583 

97.1295 

0 

71.4698 

104.0235 

137.311 

0 

320.903 

30.0158 

39.6208 

0 

01.3847 

72.72 

95.9904 

0 
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Step  2:  Select  “Total  Cost  of  Recruiting”  from  the  list  of  columns,  and  select  the 
“Y,  Response”  button. 


!*■  fit  Y  by  X  -  Contextual  -  JMP  Pro 


Distribution  of  V  for  each  X.  Modeling  types  determine  analysis. 


Select  Columns 
w  15  Columns 
^Run« 
rtr - 


atfEB 

^Education  Incentive 

^tTotal  Recruiters 

^Recruiter  Cost 

^NCO 

JlRP 

^HSDG 

^TSCI-IIIA 

^Unemployment 

^Relative  Pay 

■JQMA _ 


Cast  Selected  Columns  into  Roles 


Y  Response 

required 

X,  Factor 

required 

Block 

Weight 

Fr«| 

l*  3 

Action 

I  <*  I 


Remove 

Recall 

Help 


hi 

Bivanate  Oneway 

i\£  S3 

Logistic  Contingency 

A  A  A 


“Total  Cost  of  Recruiting”  should  now  appear  in  the  “Y,  Response”  box. 


Iy«  Fit  Y  by  X  •  Contextual  -  JMP  Pro 


Distribution  of  V  for  each  X.  Modeling  types  determine  analysis 


Select  Columns 
"  15  Columns _ 

^4 Run  » 

4fy 

^Advertising 

JEB 

^Education  Incentive 
^Total  Recruiters 
^Recruiter  Cost 
^NCO 

Jlkp 

Ahsdg 

Arsa-m 

^Unemployment 
alReiative  Pay 
l^QMA  _ 


Cast  Selected  Columns  into  Roles 


1 Y.  Response 

A  Total  Cost..  Recruiting 

|  X.  Factor 

required 

|  Block 

(  Weight 

Freq 

L_* _ J 

Action 


LZ 

Bivariat 

e 

Oneway 

L 

_ 

»gisti 

D 

Contingency 

A  i* 
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Step  3:  Select  “FY”  from  the  list  of  columns,  and  select  the  “X,  Factor”  button. 


!*■  Fit  Y  by  X  -  Contextual  *  JMP  Pro 


© 


Distribution  of  Y  for  each  X  Modeling  types  determine  analysis. 

Select  Columns  Cast  Selected  Columns  into  Roles 


•15  Columns 


1H 

Bivanate 

Oneway 

S| £ 

“6“ 

Logistic 

Contingency 

A 

£4 

Y,  Response 

A  Total  Cost-.  Recruiting 

1 

required 

Block 

Weight 

Freq 

1  By  1 

Action 
|  QIC 
Cancel 


Remove 

Recall 

Help 


“FY”  should  now  appear  in  the  “X,  Factor”  box. 


Iy«  Fit  Y  by  X  -  Contextual  -  JMP  Pro 

Distribution  of  Y  for  each  X  Modeling  types  determine  analysis. 


a 


Select  Columns 

•  15  Columns _ 

dlRun  * 

aiTotal  Cost  of  Recruiting 
^Advertising 

Ae 

Aducation  Incentive 
Aotal  Recruiters 
^Recruiter  Cost 
^NCO 

Arp 

A»sdg 

Asci-dia 

^Unemployment 
Ablative  Pay 
\JQMA  _ 


Y  Response 

A  Total  Cost..  Recruiting 

|  X,  Factor 

4.FV 

Block 

(_  Weight 

Freq 

L  By 

Action 

OK 

Cancel 


Remove 

Recall 

Help 


y 

Bivariate 

Oneway 

LC 

Logistic 

A 


Contingency 

1.4 
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Step  4:  Click-on  the  “OK”  button  to  generate  the  graph  of  FY  by  total  cost  of 
recruiting. 


Iy«  Fit  Y  by  X  -  Contextual  •  JMP  Pro 


I ,  re 


Distribution  of  Y  for  each  X  Modeling  types  determine  analysis. 
Select  Columns 
^  15  Columns 
Alun* 


Aotal  Cost  of  Recruiting 
^Advertising 

Jb  b 

Aducabon  Incentive 

aiTotal  Recruiters 

^Recruiter  Cost 

AjCO 

Arp 

Asdg 

Ascmha 

Alnemployment 

^Relative  Pay 


l£ 

Bivariate 

l*oi 

Oneway 

\L 

Logistic 

a 

Contingency 

Cast  Selected  Columns  into  Roles 
j  Y,  Response  ^ Total  Cost..  Recruiting 


X,  Factor  AfY 


Block 


Weight 

Freq 


By 


Remove 

Recall 

Help 


Step  5:  To  add  boxplots  on  the  data  for  each  FY,  select  the  red  triangle  in  the 
upper  left  hand  corner  of  the  graph.  From  the  drop-down  menu,  select 
“Quantiles.” 

lyx  JMP_129DesPts  -  Fit  Y  by  X  of  Total  Cost  of  Recruiting  by  FY  -  JMP  Pro 


pnewav  Analysis  of  T otal  Cost  of  Recruiting  By  FY 

Quantiles 

-  Shows  or  hides  a  quantile  report.  | 

Means/Anova 

»  •  * 

Means  and  Std  Dev 

:  : 

Analysis  of  Means  Methods  ► 

i  •  • 

Compare  Means  ► 

160 


B.  EXPLORE  OUTLIERS  FROM  THE  ONEWAY  ANALYSIS  GRAPH 


Step  1:  Hover  your  mouse  over  a  data  point  of  interest  to  retrieve  information 
regarding  that  point. 


4  ▼  Oneway  Analysis  of  T otal  Cost  of  Recruiting  By  FY 

1000 


900 
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700 
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Row:  78 
FY:  2015 

Total  Cost  of  Recruiting:  865.4414 
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1  row  selected 
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‘01 
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Clicking  on  the  data  point  on  the  graph  will  highlight  it  within  the  greater  data  set. 
Understanding  the  input  variables  can  help  explain  why  the  total  cost  of  recruiting 
was  unusually  high  for  this  data  point. 


Step  2:  To  explore  a  group  of  outliers,  lasso  the  data  points  of  interest  by 
creating  a  box  around  the  data  points  with  your  mouse.  Lassoing  the  data  points 
will  automatically  select  these  data  points  within  the  greater  data  set. 
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Step  3:  The  selected  data  points  can  be  further  analyzed  on  their  own.  Right- 
click  on  “Selected.” 


fe  Columns  (15/1) 

A  Run  * 

diQS 

A  Total  Cost  of  Recaiiting 
A  Advertising 
A  EB 

A  Education  Incentive 
A  Total  Recruiters 
A  Recruiter  Cost 
A  NCO 
A  LRP 
A  HSDG 
A  tsc  i-mA 
A  Unemployment 
A  Relative  Pay 
A  QMA 


»  Rows 

All  rows  303 

Selected  28 

Excluded  0 

Hidden  0 

Labelled  0 


Then  choose  “Data  View”  from  the  drop  down  menu. 
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This  will  create  a  separate  data  table  with  just  the  outliers. 


File  Edit  Tables  Rows  Cols  DOE  Analyze  Graph  Tools  View  Window  Help 


.  J  A  h  -j  ■  EB  £T«  * 


M  Untitled  15 

Run  # 

FY 

Total  Cost  of  Recruiting 

Advertising 

EB 

Education 

Linked  Subset  This  subset  is  linke 

- 

Incentive 

T< 

i 

78 

2015 

865.4414 

279.911 

369,4825 

0 

- 

2 

80 

2015 

959.1079 

321.1317 

423.8939 

0 

▼  Columns  (15/0) 

3 

88 

2015 

889.4207 

283.8923 

374.7378 

0 

A  Run  # 

4 

96 

2015 

847.7716 

263.7513 

348.1517 

0 

l  FY 

5 

207 

2016 

874.0407 

281.0028 

370.9237 

0 

A  Total  Cost  of  Recruiting 

A  Advertising 

JEB 

A  Education  Incentive 

A  Total  Recruiters 

6 

209 

2016 

968.0586 

322.3843 

425.5473 

0 

7 

217 

2016 

898.2175 

284.9996 

376.1995 

0 

8 

225 

2016 

856.4419 

264.7801 

349.5097 

0 

9 

336 

2017 

879.807 

282.0989 

372.3705 

0 

A  Recruiter  Cost 

10 

338 

2017 

974.1696 

323.6418 

427.2072 

0 

A  NCO 

11 

346 

2017 

904.2436 

286.1113 

377.6669 

0 

A  LRP 

A  HSDG 

JTSCI-mA 

A  Unemployment 

12 

354 

2017 

862362 

265.8129 

350873 

0 

13 

465 

2018 

885.6706 

283.1992 

373.8229 

0 

14 

467 

2018 

980.3789 

324.9042 

428.8735 

0 

A  Relative  Pay 

15 

475 

2018 

910.3705 

287.2273 

379.14 

0 

_ 1 

jIQMA 

16 

483 

2018 

868.3833 

266.8497 

352.2416 

0 

@  Rows 

17 

594 

2019 

891.5424 

284.3038 

375.2811 

0 

All  rows  28 

18 

596 

2019 

986.5974 

326.1715 

430.5464 

0 

Selected  28 

19 

604 

2019 

916.509 

288.3476 

380.6189 

0 

Excluded  0 

Hidden  0 

Labelled  0 

20 

612 

2019 

874.4168 

267.8906 

353.6155 

0 

21 

723 

2020 

897.9872 

285.4128 

376.7449 

0 

^  ■  "1 
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C.  SELECT  ONE  FY  TO  ANALYZE 

To  focus  analysis  on  one  specific  FY,  the  other  six  FYs  must  be  hidden  and 
excluded.  In  this  example,  FY  2017  is  the  FY  of  interest.  FYs  2015,  2016,  2018, 
2019,  2020,  and  2021  will  be  hidden  and  excluded. 

Step  1:  To  exclude  FY  2015  and  2016,  select  on  the  first  row  of  FY  2015  data  in 
the  furthest  column  to  the  left.  Hold  the  “shift”  keyboard  button. 


Eg  JMP_1290esPts  -  JMP  Pro 

File  Edit  Tables  Rows 

Cols  DOE 

Analyze 

Graph 

Tools  View  Window  Help 

/ 

_ 

LSI0J  t  -  O 

<3® 

a  e-c.  * 

-  JMP_129DesPts 

0 

•  ■ 

Total  Cost  of 

Kecruitmg  Advertising 

1 

2015 

302.7718 

25.2367 

2 

2 

2015 

364.2534 

50.3606 

- 3 — 

- 3— 

-29*5- 

407.1973 

84.0512 

4 

4 

2015 

272668 

15.5794 

5 

5 

2015 

314.7028 

24.5896 

iFY 

6 

6 

2015 

455.8674 

80.494 

A  Total  Cost  of  Recruiting 

7 

7 

2015 

296.7907 

9.7023 

A  Advertising 

8 

8 

2015 

461.4439 

75.9782 

A  EB 

9 

9 

2015 

386.4563 

66.0029 

10 

10 

2015 

410.5732 

77.775 

11 

11 

2015 

314.3641 

27.1973 

A  NCO 

12 

12 

2015 

343.2203 

47.6491 

A  LRP 

13 

13 

2015 

309.9515 

8.491 

A  HSDG 

14 

14 

2015 

488.9989 

86.761 

A  Unemployment 

15 

15 

2015 

309.0097 

7.2378 

16 

16 

2015 

423.4232 

56.3068 

A  QMA 

17 

17 

2015 

424451 

84.3216 

18 

18 

2015 

377506 

61.8626 

19 

19 

2015 

299.8877 

27.8769 

20 

20 

2015 

324.3671 

34.8274 

21 

21 

2015 

331.2071 

22.3482 

22 

22 

2015 

427.W95 

69.7317 

»  Rows 

23 

23 

2015 

290.6615 

6.5309 

All  rows  903 

24 

24 

2015 

407.0701 

49.7876 

25 

25 

2015 

316279 

2887 

0 

26 

26 

2015 

422.3103 

74.2907 

Labelled 

0 

27 

27 

2015 

301.4469 

28.8314 

*  \7Z 

evaluations  done 
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Step  2:  Scroll  down  to  the  last  row  of  FY  2016  data  (which  appears  in  row  “258”). 
Click  on  the  “258”  cell  in  the  furthest  column  to  the  left. 


Eii  JMP_1290esPts  -  JMP  Pro 

We  Ed«t  TaWes  Rows 

a  B  -*  * 

| »  JMP_129QesPt» 
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245 
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15.7012 
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325.5253 

24  7818 
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264 

2017 
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811232 

Step  3:  Right-click  on  the  selected  rows,  and  choose  “Hide  and  Exclude”  from 
the  drop  down  menu. 


ESI  JMP_129DesPts  -  JMP  Pro 

File  Edit  Tables  Rows  Cob  DOE  Analyze  Graph  Toob  View  Window  Help 
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You  should  now  see 
2016. 


©6i 


next  to  each  row  of  data  from  FY  2015  and 


ES  JMP_129DesPt5  -  JMP  Pro 


File  Edit  Tables  Rows  Cob  DOE  Analyze  Graph 

Tools 

View  Window  Help 

e  a  A  ft  -  j 

tt-)  Hi  tr*  * 

-  JMP  129D«Pts 

Total  Cost  of 

-  Source 

JC 

FY 

Recruiting 

Advertising 

248 

2016 

447.4629 

95.7411 

249 

2016 

348.5629 

39.4425 

®» 

250 

2016 

597.7795 

157.4634 

•  Columns  (15/1) 

Gao 
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11.1203 
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43.0785 
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15.7012 

263 
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81.1232 

Step  4:  Follow  steps  1-3  to  hide  and  exclude  data  from  FY  2018,  2019,  2020 
and  2021 .  Row  388  is  the  first  row  of  data  for  FY  201 8. 
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»  Rows 


All  rows 
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evaluations  done 


167 


D.  DISTRIBUTION 

The  distribution  of  the  total  cost  of  recruiting  for  FY  2017  is  explored.  This 
technique  can  be  applied  to  any  of  the  output  variables  to  better  understand  its 
distribution  and  possible  spread  values. 


■*  Distributions 


A  •»  Total  Cost  of  Recruiting 


1— TT^l - M . 

— i 

hi'1  -  ,  r-r-r— i 

.  .... 
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99.5% 

97.5% 
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0.5% 
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0.0% 
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A 
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974.1696 
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398.24689 

974.1696 

Std  Dev 

124.63855 

875.44575 

Std  Err  Mean 

10.973812 

533.85 

Upper  95%  Mean 

419.96045 

433.12145 

Lower  95%  Mean 

376.53333 

361.2135 

N 

129 

320.18825 

293.5959 

276.82095 

261.6095 

261.6095 


Step  1:  Select  “Analyze”  from  the  ribbon,  and  select  “Distribution”  from  the  drop 
down  menu. 


JMP  129DesPts  -  JMP  Pro 


File  Edit  Tables  Rows  Cols  DOE  Analyze  Graph  Tools  View  Window  Help 


A\ 

ria  FY 

A  Total  Cost  of  Recruiting 
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Step  2:  Select  “Total  Cost  of  Recruiting”  from  the  list  of  columns,  and  click  on  the 
“Y,  Columns”  button. 


1st  Distribution  -  JMP  Pro 
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The  distribution  of  values  in  each  column 
Select  Columns 

'Til  5  Columns _ 
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Action 
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required 
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[  Weight  | 

optional  numeric 

Freq 

1  By  1 

optional 

OK 


Cancel 


Remove 

Recall 

Help 


The  distribution  for  Total  Cost  of  Recruiting  will  appear. 

Step  3:  To  rotate  the  distribution  to  appear  horizontal,  click  on  the  red  triangle  in 
the  upper  left  hand  corner  of  the  graph,  and  select  “Stack”  from  the  drop  down 
menu. 
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E.  PARTITION  TREES 

The  partition  tree  on  total  cost  of  recruiting  will  be  explored.  The  partition  tree  is  a 
useful  method  that  can  help  provide  insights  into  variable  interactions. 


Step  1:  To  create  a  partition  tree,  select  “Analyze”  from  the  ribbon.  Then  choose 
“Modeling,”  and  “Partition”  from  the  drop  down  menus. 


EBB  JMP 129DesPts  -  JMP  Pro 


File  Edit  Tables  Rows 


’  JMP 129DesPts 


Source 


Cols  DOE 

^  a87 
yj  < 


Analyze  Graph  Tools  View  Window  Help 


Columns  (15/1) 

Z0EE 

*FY 

A  Total  Cost  of  Recruiting 
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regression  trees. 
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Step  2:  Select  “Total  Cost  of  Recruiting”  from  the  list  of  columns,  and  click  on  the 
“Y,  Response”  button. 
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Remove 

Recall 
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Step  3:  Select  each  decision  variable  (Advertising,  EB,  Education  Incentive, 
Total  Recruiters)  from  the  list  of  columns,  then  click  on  the  “X,  Factor”  button. 
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Step  4:  Click  on  the  “OK”  button. 
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The  partition  tree  window  will  pop-up  with  just  the  parent  node. 
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Step  5:  To  make  the  first  split  on  “Total  Cost  of  Recruiting,”  click  on  the  “Split” 
button. 
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Continue  to  split,  by  clicking  the  “Split”  button.  If  you  want  to  undo  a  split,  click  on 
the  “Prune”  button.  A  “Training”  R2  value  of  0.80  is  an  adequate  threshold  to 
achieve.  In  this  case,  disregard  the  “Validation”  R2  value. 
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F.  STEPWISE  REGRESSION  MODEL 

To  develop  a  model  for  the  total  cost  of  recruiting,  stepwise  regression  is  used  to 
determine  the  beta  estimates  to  fit  a  model. 

Step  1 :  Select  “Analyze”  from  the  ribbon,  then  “Fit  Model”  from  the  drop  down 
menu. 
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Step  2:  Select  “Total  Cost  of  Recruiting”  from  the  list  of  columns,  and  click  on  the 
“Y”  button. 
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Step  3:  While  holding  the  Ctrl  key,  select  each  market  factor  that  was  varied  in 
the  PROM-WED  excursion. 
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Select  the  right  corner  of  the  “Macros”  button  (i.e.,  the  arrow),  and  select 
“Factorial  to  degree”  from  the  drop-down  menu. 
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degree.  E.g.,  degree  2  enters  mam 
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This  will  add  all  main  effect  and  two-way  interactions. 
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Again,  while  holding  the  Ctrl  key,  select  each  market  factor  that  was  varied  in  the 
PROM-WED  excursion.  Select  the  right  corner  of  the  “Macros”  button  (i.e.,  the 
arrow),  and  select  “Polynomial  to  degree”  from  the  drop-down  menu.  This  will 
add  all  second  degree  polynomial  interactions. 
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Step  4:  From  the  “Personality”  drop-down  menu,  select  “Stepwise. 
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optional  numeric 

Validation 

optional 

By 


option 


Construct  Model  Effects 
NCO 


Add 


Cross 


Nest 


Macros  ▼ 


Degree  |  2  | 
Attributes  ▼ 
Transform  t 
El  No  Intercept 


Personality:  (standard  Least  Squares 


Emphasis: 


Help 


Recall 


Remove 


Unemployment 
Relative  Pay 
NCO’Unemployment 
NCO* Relative  Pay 
Unemployment’ Relative  Pay 
NCO*NCO 

Unemployment’Unemployment 
Relative  Pay’Relative  Pay 


Mixed  Model 
Manova 

Loglinear  Variance 

Nominal  Logistic 
Ordinal  Logistic 


Proportional  Hazard 
Parametric  Survival 

Generalized  Linear  Model 


Partial  Least  Squares 


Response  Screening 


Step  5:  Ensure  that  the  “Keep  dialog  open”  box  is  checked,  and  click  the  “Run” 
button. 


A  »  Model  Specification 

Select  Columns 
^15  Columns 
^Run  * 

Af'i 

Aotal  Cost  of  Recruiting 
^Advertising 

AB 

^Education  Incentive 
A~ ota I  Recruiters 
^Recruiter  Cost 
AlCO 

Arp 

Aisdg 

Asci-iiia 

^Unemployment 
^Relative  Pay 
iQMA 


Pick  Role  Variables 


(  V  1 

^ Total  Cost  of  Recruiting 

optional 

Weight  ] 

|  optionalnumeric 

[  F'e<i  I 

[optional  numeric  \ 

Validation 

|  optional  \ 

Personality:  |  Stepwise 


Help 

Run 

Recall 

J  Keep  dialog  open 

Remove 

By  ||  optional 


Construct  Model  Effects 


(  Add  1 

NCO 

|  -  | 

Unemployment 

Cross 

Relative  Pay 

Nest 

NCC’Unemployment 

NCO*  Relative  Pay 

Macros  ▼ 

Unemployment'Relative  Pay 

Degree  |  2] 

NCO'NCO 

Unemployment’Unemployment 

Transform  y 

No  Intercept 

Relative  Pay'Relative  Pay 

3 
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Step  6:  The  “Stepwise  Regression  Control”  window  will  appear.  Press  the  “Go” 
button. 


d  *  Stepwise  Fit  for  Total  Cost  of  Recruiting 
i  Stepwise  Regression  Control 


Stopping  Rule;  |  Minimum  BIC 
Direction;  Forward  - 


@ 

f  Enter  All  Make  Model 

Remove  All  Run  Model 

|  Combine  * 


5 

Stop 

Step  J 

//4  rows  no1 

used  due  to  excluded  rows  or  missing  values. 

SSE 

DFE  RMSE  RSquare  RSquare  Adj 

Cp 

p  AICc 

BIC 

19884503 

128  124.63855  0.0000  0.0000  717.81945 

1  1614.135 

1619.76 

2i  Current  Estimates 

Lock 

Entered  Parameter 

Estimate 

nDF 

SS 

"F  Ratio" 

"Prob>F- 

Intercept 

398.246889 

1 

0 

0.000 

1 

NCO 

0 

1 

7201112 

72.105 

4.6e-14 

□ 

Unemployment 

0 

1 

108286.7 

7.314 

0.00778 

D 

G 

Relative  Pay 

0 

1 

533851.6 

46.610 

3.2e-10 

□ 

G 

(NCO-35000.1)"(Unemployment-60031) 

0 

3 

8415882 

30.576 

6.8e-15 

□ 

(NCO-35000.1)'(Relative  Pay-1) 

0 

3 

1459125 

114.857 

9.2e-36 

(Unemployment-6.003 l)"(Relative  Pay-1) 

0 

3 

657070.1 

20.564 

6.8e-U 

E 

G 

(NCO-35000.1)'(NCO-35000iD 

0 

2 

8376808 

45.860 

l.le-15 

□ 

G 

(Unemployment-6.003inUnemployment-6.0031) 

0 

2 

1109264 

3.722 

0.02688 

□ 

G 

(Relative  Pay-l)*(Relative  Pay-1) 

0 

2 

5423772 

23.629 

1.93e-9 

Step  7:  Once  settled,  select  the  “Run  Model”  button. 


A 


-  Stepwise  Fit  for  Total  Cost  of  Recruiting 

me  j 

_  Stepwise  Regression  Control 

Stopping  Rule;  Minimum  BIC  -  *  Enter  AN  1  Make  Model 

Direction:  Forward  »  4"  Remove  All 

Run  Model 

|  Combine  w  ]  |  Take  current  model  and  fit  it  in  a  separate  full-featured  fitting  platform. 


6°  Stop  Step 

774  rows  not  used  due  to  excluded  rows  or  missing  values. 


SSE  DFE  RMSE  RSquare  RSquare  Adj  Cp  p  AlCc  BIC 

286974.74  120  48  902517  0.8557  0.8461  10.92502  9  1382.197  1408.931 


^  Current  Estimates 


Lock 

Entered 

Parameter 

Estimate 

nDF 

SS 

"F  Ratio" 

"Prob>F" 

Intercept 

150.492117 

1 

0 

0.000 

1 

□ 

13) 

NCO 

0.02552672 

4 

1034475 

108.143 

7.7e-39 

o 

13! 

Unemployment 

-24.598171 

3 

132374 1 

18.451 

6.6e-10 

□ 

13) 

Relative  Pay 

•550.24982 

4 

7341281 

76.745 

3.7e-32 

m 

(NCO-35000.1)*(Unemployment-60031) 

-0.0035413 

1 

1606216 

6.716 

0.01074 

E 

D 

(NCO-3  5000.  l)*(Relative  Pay-1) 

-0.1231431 

1 

1638472 

68.514 

2e-13 

D 

13) 

(Unemployment-6.0031)’(Relative  Pay-1) 

84.0662002 

1 

15660.78 

6.549 

0.01174 

D 

13) 

(NCO-35000.1)*(NCO-35000.1) 

3.09283e-6 

1 

6278413 

26.254 

116e-6 

□ 

E 

(Unemployment-6.0031)’(Unemployment-6.0031) 

0 

1 

6884.615 

2.925 

0.08982 

E 

(3) 

(Relative  Pay-l)"(Relative  Pay-1) 

1907.37931 

1 

6309642 

26.384 

l.le-6 

a  Step  History 

Step  Parameter  Action 

1  {NCO-35000.1)*(Rdative  Pay-1)  Entered 

2  Unemployment  Entered 

3  (NCO-35000.1)’(NCO-3500ai)  Entered 

4  (Relative  Pay-l)*(Relative  Pay*l)  Entered 

5  (NCO-35000.1)’(Unemployment-60031)  Entered 

6  (Unemployment*6.0031)*(Relatrve  Pay-1)  Entered 

7  (Unemployment-6.0031)*(Unemployment-6.0031)  Entered 

8  Best  Specific 


"SigProb"  SeqSS 

0.0000  1459125 
0.0000  105529.6 
0.0001  48541.67 
0.0000  61580.54 
0.0377  1103826 
0.0117  15660.78 
0.0898  6884.615 


RSquare  Cp 

0.7338  103.89 
0  7869  61.055 
0.8113  42.432 
0.8423  18.268 
0.8478  15.579 
0.8557  10.925 
0.8591  10 

0.8557  10.925 


p  AICc  BIC 

4  1449.8  146381  0 

5  142331  1439.78  0 

6  140986  142895  0 

7  1389.01  141089  0 

8  1386.7  141093  O 

9  13822  140893  O 

10  138146  141086 O 

9  13822  140893® 
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The  “Report:  Fit  Model”  window  will  appear. 


Report:  Fit  Model  -  JMP  Pro 


a  ▼  Fit  Group 

A  ▼  Response  Total  Cost  of  Recruiting 
A  Effect  Summary 

Source 

NCO 

Relative  Pay 
NCO’Relative  Pay 
Unemployment 
Relative  Pay’Relative  Pay 
NCO*NCO 

NCO’Unemployment 
Unemployment*Relative  Pay 

Remove  Add  Edit  IH  FDR 

A  Summary  of  Fit 

RSquare  0.855679 

RSquare  Adj  0.846058 

Root  Mean  Square  Error  48.90252 

Mean  of  Response  398.2469 

Observations  (or  Sum  Wgts)  129 

A  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square 

Model  8  1701475.6  212684 

Error  120  286974.7  2391 

C  Total  128  19884503 

A  Parameter  Estimates 

Term  Estimate  Std  Error  t  Ratio  Prob>|t| 


Intercept 

150.49212 

68.20602 

2.21  0.0293* 

NCO 

0.0255267 

0.00148 

17.25  <.0001* 

Unemployment 

-24.59817 

3.699755 

-6.65  <.0001* 

Relative  Pay 

-550.2498 

37  00096 

-14.87  <.0001* 

(NCO-35000.1)*(Unemployment-6.0031) 

-0.003541 

0.001366 

-2.59  0.0107* 

(NCO-35000.1)*(Relative  Pay-1) 

-0.123143 

0.014877 

-8.28  <.0001* 

(Unemployment-6.0031)’(Relative  Pay-1) 

84.0662 

32.8508 

2.56  0.0117* 

(NCO-35000.1)*(NCO-3  5000.1) 

3.0928e-6 

6.036e-7 

5.12  <.0001* 

(Relative  Pay-l)*(Relative  Pay-1) 

1907.3793 

371335 

5.14  <.0001* 

F  Ratio 

88.9351 
Prob  >  F 

<.0001* 


LogWorth  PValue 

0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.01074 
0.01174 


At  this  point,  you  can  decide  if  you  would  like  to  make  manual  adjustments  to  the 
stepwise  regression.  For  example,  the  interactions  between  unemployment  rate 
and  relative  pay,  and  the  new  accession  mission  and  unemployment  in  this 
example  both  exhibit  low  “t  Ratio”  values. 
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To  remove  these  terms  from  the  model,  return  to  the  “Stepwise  Fit”  window,  and 
uncheck  the  terms  in  the  “Entered”  column  that  you  would  like  to  remove.  Select 
“Run  Model”  to  fit  the  new  model. 


•  Stepwise  Fit  for  Total  Cost  of  Recruiting 

A  Stepwise  Regression  Control 

Stopping  Rule  Minimum  BIC  »  ■»  EnteiAII 

Make  Model 

Direction 

1  Forward  *  |  |  #  |  [Remove  Ml 

Run  Model 

Rules: 

Combine  * 

(_  Go 

_[[  Stop  [  Step 

774  rows  not  used  due  to  excluded  rows  or  missing  values. 

SSE 

DFE  RMSE  RSquare  RSquare  Adj  Cp 

p  AICc 

BIC 

3030365 

121  50.044352  0.8476  0.8388  15.749241 

8  1386.871 

1411.096 

A  Current  Estimates 

Lock  Entered  Parameter 

Estimate  nDF 

ss 

"F  Ratio" 

"Prob>F" 

Intercept  151.602299 

1 

0 

0.000 

1 

□  m 

NCO  0.02552971 

3 

1018413 

135.548 

1.6e-38 

□  W 

Unemployment 

-24.567887 

2 

116311.9 

23.221 

2.92e-9 

3  r^—\ 

Relative  Pay 

•550.43355 

4 

7423115 

74.100 

1.3e-31 

(NCO-35000.1)'(Unemployment-6.0031) 

0 

1 

16062.16 

6.716 

0.01074 

□  1 - 

(NCO*35000.1)’(Relative  Pay-1) 

-0.1277552 

1 

1789104 

71.437 

7.6e-14 

(Unemployment-6.0031),(Relative  Pay-1)  68.0443421 

1 

10636,88 

4.247 

0.04146 

L-o-1 

(NCO-35000.1)*(NCO-350G01) 

2.93986e-6 

1 

57275 

22.869 

4.94*6 

(Unemployment-6.0031)’(Unemployment-6.0031) 

0 

1 

9014024 

3.679 

0.05748 

□  E0 

(Relative  Pay-l)*(Relative  Pay-1)  1917.11709 

1 

63748.85 

25.454 

1.61e-6 

A  Step  History 

Step 

Parameter 

Action 

-Sig  Prob" 

SeqSS 

RSquare 

Cp 

P 

AICc 

BIC 

1 

(NCO-35000.1)*(Relative  Pay-1) 

Entered 

0.0000 

1459125 

0.7338 

103.89 

4 

1449.8 

146361 

2 

Unemployment 

Entered 

0.0000 

1055296 

0.7869 

61.055 

5 

142331 

1439.78 

3 

(NCO-35000.1)*(NCO-35000.1) 

Entered 

0.0001 

4^ 

03 

£ 

S 

0.8113 

42.432 

6 

140956 

1428.95 

4 

(Relative  Pay-l)*(Relative  Pay-1) 

Entered 

:  OOOQ 

6158054 

0.8423 

18  268 

7 

138901 

1410.69 

5 

(NCO-35000.1)*(Unemployment-60031) 

Entered 

0.0377 

1103826 

0.8478 

15.579 

8 

1386.7 

1410.93 

6 

(Unemployment-6003 1 ) * ( Relative  Pay- 1) 

Entered 

0.0117 

15660.78 

0.8557 

10.925 

9 

13822 

1408.93 

7 

(Unemployment-6.0031)’(Unemployment-6.0031)  Entered 

0.0898 

6884.615 

0.8591 

10 

10 

138146 

141066 

8 

Best 

Specific 

0.8557 

10.925 

9 

13822 

140893 

9 

(NCO-35000.1)*(Unemployment-60031) 

Removed 

0.0107 

1606216 

0.8476 

15.749 

8 

138637 

14111 

Step  8:  To  graph  the  “Actual  by  Predicted”  plot,  select  the  red  triangle  next  to 
“Response  Total  Cost  of  Recruiting.”  From  the  drop-down  menu,  select  “Row 
Diagnostics”  and  “Plot  Actual  by  Predicted.” 


0V  Report:  Fit  Model  -  JMP  Pro 


”  Fit  Group 
*  j  :  sDonsc  T otal  Cost  of  Recruiting 


Regression  Reports  ► 

Estimates  ► 

Effect  Screening  ► 

Factor  Profilina  ► 

g  Worth 

33.553  | 

38  2731 1  - 

Row  Diagnostics 

Plot  Actual  by  Predicted 

Save  Columns  * 

Plot  Effect  Leverage 

Model  Dialog 

Plot  Residual  by  Predicted 

✓  Effect  Summary 

Plot  Residual  by  Row 

Script  ► 

Press 

Durbin  Watson  Test 

A  oBbCcDt  Aai 
iubJle  Em,..  Err 


PValue 

0.00000 

0.00000 


Actual  response  value  on  Y  axis,  by 
Predicted  value  on  the  X  axis.  In  good 
fits,  points  are  near  the  diagonal.  You 
can  see  which  points  do  not  fit  look 
for  patterns,  visualize  the  test. 
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Step  9:  To  fit  the  prediction  model,  select  the  red  triangle  next  to  “Response 
Total  Cost  of  Recruiting.”  From  the  drop-down  menu,  select  “Estimates”  then 
“Show  Prediction  Expression.” 


f,  Report  Fit  Model  •  JMP  Pro 


4  »  Fit  Group 


i  esDonse  Total  Cost  of  Recruiting 


»nrw:inn  Reports 


Estimates 


fcnea  screening 

Factor  Profiling 
Row  Diagnostics 
Save  Columns 

Model  Dialog 
Effect  Summary 
Script 


Show  Prediction  Expression 


•-  1  ~ 

Expanded  Estimates 
Sequential  Tests 
Custom  Test 
Multiple  Comparisons 
Joint  Factor  Tests 
Inverse  Prediction... 


Displays  or  hides  the  prediction 
formula  in  the  report. 


trtfiriXxj 

a  ooooo 
0.00000 
0.00000 
0.00000 
0.00000 
0.01074 
0.01174 
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G.  SCATTERPLOT  MATRIX 

Scatterplot  matrices  can  be  used  to  visualize  trends  when  multiple  variables  are 
changing. 


Step  1:  Select  “Graph”  from  the  ribbon,  then  “Scatterplot  Matrix”  from  the  drop 
down  menu. 


HI  JMP_129DesPts  -  JMP  Pro 


File  Edit  Tables  Rows  Cols  DOE  Analyze 

e3  0  ka‘  A 

E®  E0  E?-  It 

-  JMP 129D«Pts 

▼  Source 

\w\ 

oa  878 

O  St  879 

oa  880 

*  Columns  (15/1) 

OH  881 

oa  882 

i,FY 

sa  883 

A  Total  Cost  of  Recruiting 

A  Arivertkinn 

oa  884 

Graph 


Tools  View  Window  Help 


Bubble  Plot 


Scatterplot  Matrix 


*S|j  Parallel  Plot 
v:;  Cell  Plot 

:£-  Scatterplot  3D 
%  Contour  Plot 
A  Ternary  Plot 


Educatn 


H  Displays  multivariate  data  in  a  grid  of 

I  2-dimensional  scatterplots. 

)ti  1  Jr.uwy 

•3 
19 
>2 

II 
>5 


16.9685  22.3984 

113.5786  149.9238 
13.87  18.3085 

81.1356  107.099 

31.5034  41.5845 


nc  ruin. i 
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Step  2:  To  set  the  Y-axis  variables,  select  “Total  Cost  of  Recruiting”  and 
“Advertising”  from  the  list  of  columns,  and  click  on  the  “Y,  Columns”  button. 


.Jeb - 

jlEducation  Incentive 

^Total  Recruiters 

^Recruiter  Cost 

^NCO 

.Jlrp 

jJhsdg 

^TSCWDA 
^Unemployment 
^Relative  Pay 
'.JQMA 


Scatterplots  of  all  pairs  of  Y  variables  or  all  X-Y  pairs  if  X's  specified 

Select  Columns  -  - ,  Cast  Selected  Columns  into  Roles 

fel5  Columns 
^Run  # 

Aey - 


Y,  Columns 


required 


option, 


Group 


By 


Matrix  Format  Lower  Triangular  * 


Action 


OK 


Cancel 


Remove 


Recall 


Help 


Step  3:  To  set  the  X-axis  variables,  select  the  variables  of  interest  (NCO, 
Unemployment  Rate  and  Relative  Pay  in  this  case),  and  click  on  the  “X”  button. 


Scatterplots  of  all  pairs  of  Y  variables,  or  all  X-Y  pairs  if  X’s  specified 


^Run  # 

iFY 

^Total  Cost  of  Recruiting 

^Advertising 

^EB 

^Education  Incentive 
^Total  Recruiters 

^Recruiter  Cost - 

4 _ 

altftP - 

Jhsdg 

^TSCI-mA 
^Unemployment 
^Relative  Pay 
I^QMA 


Cast  Selected  Columns  into  Roles 


Y,  Columns 

^Total  Cost...  Recruiting 
^Advertising 

optional 

X 

optional 

Group 

optional 

[  By 

’ 

Select  Columns 
-  15  Columns 


Matrix  Format  |  Lower  Triangular  •* 


Action 

OK 

Cancel 


Remove 

Recall 

Help 


I'l- 


Step  4:  Repeat  Step  3  for  Unemployment  Rate  and  Relative  Pay. 
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Step  5:  To  generate  the  scatterplot  matrix,  click  the  “OK”  button. 


4i-  Scatterplot  Matrix  -  JMP  Pro 


I 


Scatterplots  of  all  pairs  of  Y  variables  or  all  X-Y  pairs  if  X's  specified 


Select  Columns 

@15  Columns _ 

jlRun  * 

Afy 

^Total  Cost  of  Recruiting 
^Advertising 

AtB 

^Education  Incentive 
^Total  Recruiters 
^Recruiter  Cost 
^NCO 

Arp 

Jhsdg 

ASCI-IIIA 


^QMA 


Matrix  Format 


Lower  Triangular  » 


Cast  Selected  Columns  into  Roles 


Y,  Columns 

^  Total  Cost...  Recruiting 
^Advertising 

optional 

X 

AHCO 

^Unemployment 

A  Relative  Pay 

optional 

Group 

optional 

By 

Action 


* 


Step  6:  To  fit  a  trend  line  on  the  plots,  click  the  red  triangle,  and  select  “Fit  Line” 
from  the  drop  down  menu. 


-^Scatterplot  Matrix 

✓  Show  Points  I 


Fit  Line 

Density  Ellipses 

Nonpar  Density 
Group  By... 

Lock  Scales 
Script 

300 

250 


Displays  or  hides  the  regression  line 
and  confidence  interval  on  the 
scatterplot  matrix. 


►•A  •  •  • 

V%  •  . 

F  •  ».*•••  •  *•  *•  »*  ;«k  r  • 

’  .ly. 


i 


< 


200 

• 

• 

• 

• 

• 

• 

150 

•  • 

•* 

*  • 

..  V/ 

*  ,  * 

100 

50 

0 

••  • 

•  •  •*.  • 

•  VA  T 

31000  34000  37000 

4  45  5  5  5  6  65  7  75 

06  0.9  1  1.1 

Relative 

NCO  Unemployment  Pay 
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H. 


CONTOUR  PLOTS 


▼  Contour  Plot  for  Total  Cost  of  Recruiting 

40000 
39000 
38000 
37000 
36000 
NCO  35000 
34000 
33000 
32000 
31000 
30000 


0.80  0.85  0.90  0.95  1.00  1.05  1.10  1.15  1.20 


Relative  Pay 


Total  Cost  of  Recruiting 

■  <=  300.0 
■i  <=  400.0 

■  <=  500.0 

■i  <=  600.0 

■  <=  700.0 
■■  <=  800.0 

■  <=900.0 

■  >  900.0 


Step  1:  Select  “Graph”  from  the  ribbon,  then  “Contour  Plot”  from  the  drop  down 
menu. 


EH  JMP_129DesPts  -  JMP  Pro 


File  Edit  Tables  Rows  Cols  DOE  Analyze 


Graph 


Tools 


View 


Window 


Help 


da  £)  1*  A 

EE  !=-  £ 

PH  Graph  Builder 

1 _ 

-  JMP 129DesPts 

Bubble  Plot 

Educ 

T  Source 

▼ 

Ri 

tjij.  Scatterplot  Matrix 

Advertising 

EB 

Ince 

®«  878 

w 

90.5324 

119.5028 

Oii  879 

[jjjfj  Parallel  Plot 

)8 

43.8552 

57.8889 

0(3  880 

ceiipiot 

16.9685 

22.3984 

■»  Columns  (15/1) 

0(3  881 

Scatterplot  3D 

113.5786 

149.9238 

OA  882 

)2 

13.87 

18.3085 

1  fy 

0(3  883 

>'  Contour  Plot 

- 1 

l  Displays  three  variables  in  a  2- 

A  Total  Cost  of  Recruiting 

0(3  884 

A  Ternary  Plot 

j  dimensional  view  where  the  third 

A  Advertising 

0(3  885 

[.  variable  is  represented  by  contour 

►  _ L _ 

A  EB 

0(3  886 

Surface  Plot 

M  curves  ui  cqudi  vdiuc. 

_ 1 
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Step  2:  To  set  “Total  Cost  of  Recruiting”  as  the  variable  represented  by  the  color 
scale,  select  “Total  Cost  of  Recruiting”  from  the  list  of  columns,  and  click  the  “Y” 
button. 


Please  specify  two  X  columns  and  one  or  more  Y  columns. 


Select  Columns 
w  15  Columns 
^Run  * 
'•FY- 


4eb 

^Education  Incentive 

^Tctal  Recaiiters 

^Recruiter  Cost 

^NCO 

^LRP 

^HSDG 

Arse  i-mA 

^Unemployment 
^Relative  Pay 
^QMA 


Options 
Contour  Values: 

Display: 

Data: 


Cast  Selected  Columns  into  Roles 


j  Fill  Areas 
y  Use  Table  Data 


Action 


V 

required  numeric 

[  X  J 

required  numeric 
required  numeric 

L  By  J 

OK 


Cancel 


Remove 

Recall 

Help 


Step  3:  To  set  “Relative  Pay”  as  the  x-axis,  select  “Relative  Pay”  from  the  list  of 
columns,  and  click  the  “X”  button. 


Please  specify  two  X  columns  and  one  or  more  Y  columns. 

Select  Columns  fact  MprfpH  fnlumnc  intn  Rnl«  Artinn 


15  Columns 
^Run  * 

Afy 

^Total  Cost  of  Recruiting 
^Advertising 

Jeb 

^Education  Incentive 
^#Total  Recruiters 
^Recruiter  Cost 
ANCO 
Air? 

Ahsdg 

a^TSC  I-mA 


Options 

Contour  Values:  (  Specify... 
Display:  O  Fill  Areas 

Data:  g]  Use  Table  Data 
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Step  4:  To  set  the  new  accession  mission  (NCO)  as  the  y-axis,  select  “NCO” 
from  the  list  of  columns,  and  click  the  “X”  button. 


Step  5:  Select  the  “Fill  Areas”  box,  then  click  the  “OK”  button  to  generate  the 
contour  plot. 


Please  specify  two  X  columns  and  one  or  more  Y  columns. 


Select  Columns 
w  15  Columns 
^iRun  # 

Afy 

^iTotal  Cost  of  Recruiting 

^Advertising 

JEB 

^Education  Incentive 
^Total  Recruiters 
^Recruiter  Cost 


.NCO 


ARP 

I^HSDG 

•Jtsci-iha 

^Unemployment 
^Relative  Pay 

l^QMA 

Options 


Cast  Selected  Columns  into  Roles 


1  v  I 

A  Total  Cost...  Recruiting 

1  X  1 

A  Relative  Pay 

JNCO 

1  By  I 

Action 


Contour  Values: 

Specify... 

|  |  Retneve... 

Display: 

J_  Fill  Areas 

Data: 

1 V,  Use  Table  091a  1  Specify  Grid... 
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V. 


EXAMPLE  TEST  CASES 


Two  test  case  examples  are  provided  to  demonstrate  PROM-WED’s 
capabilities. 

A.  EFFECT  OF  ECONOMIC  UNCERTAINTIES 

What  is  the  optimal  allocation  of  recruiting  resources  that  is  robust  to  a 
broad  range  of  economic  uncertainties? 


Variable  Type 

Variable  Name 

Value  Low 

Value  High 

Decision  Variable 

Recruiters 

2,500  recruiters 

3,500  recruiters 

Market  Factor 

Unemployment  Rate 

4.0% 

8.0% 

Market  Factor 

Relative  Pay 

0.80 

1.20 

Policy  Factor 

Recruiting  Mission  (NCO) 

30,000  recruits 

40,000  recruits 

B.  EFFECT  OF  LEGALIZATION  OF  MARIJUANA  TEST  CASE: 

What  is  the  optimal  allocation  of  recruiting  resources  if  the  Navy  desires  to 
increase  the  percentage  of  high  quality  recruits  from  70  percent  to  85  percent? 
Due  to  uncertainties  in  the  current  fiscal  environment,  the  unemployment  rate 
may  fluctuate  between  4  to  8  percent  and  the  ratio  of  relative  pay  may  vary 
between  0.8  and  1.2.  In  addition,  since  marijuana  has  been  legalized  for 
recreational  use  in  many  states  nationwide,  drug-use  amongst  18-24  year-olds  is 
expected  to  increase.  An  increase  in  drug-use  means  less  young  adults  qualify 
for  military  service.  This  test  case  models  the  effect  of  an  annual  decrease  of 
10,000  qualified  military  available  due  to  pre-service  drug-use. 


Variable  Name 

Value  Low 

Decision  Variable 

Production  Recruiters 

2,500 

recruiters 

3,500  recruiters 

Market  Factor 

Unemployment  Rate  (UE) 

4.0% 

8.0% 

Market  Factor 

Percentage  of  High  Quality 

Recruits  (TSC  l-lll) 

70% 

85% 

Market  Factor 

Relative  Pay 

0.8 

1.2 

Market  Factor 

Qualified  Military  Available 
(QMA) 

*See  Table  13 

Policy  Factor 

Recruiting  Mission  (NCO) 

30,000  recruits 

40,000  recruits 
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Cumulative  Effect  of  Decrease  in  QMA 


FY 

QMA  Value  Low 

QMA  Value  High 

2015 

1,873,304 

1,883,304 

2016 

1,863,304 

1,873,304 

2017 

1,853,304 

1,863,304 

2018 

1,843,304 

1,853,304 

2019 

1,833,304 

1,843,304 

2020 

1,823,304 

1,833,304 

2021 

1,813,304 

1,823,304 
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APPENDIX  D.  SCENARIO  INPUT  REPORTS 


A.  PARAMETER  INPUTS  FOR  FIGURE  45 

Where  “Recruiters”  is  the  only  variable  that  is  fixed.  EB,  NCF,  and 


advertising  are  floated. 


2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCF 

high 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

low 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

UE 

high 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

low 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

Recruiters 

high 

3913 

3913 

3913 

3913 

3913 

3913 

3913 

low 

3913 

3913 

3913 

3913 

3913 

3913 

3913 

LRP 

high 

7.44 

11.22 

11.28 

11.38 

11.43 

11.46 

11.67 

low 

7.44 

11.22 

11.28 

11.38 

11.43 

11.46 

11.67 

Advertising 

high 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

low 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

EB 

high 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

low 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

NCO 

high 

35025 

36425 

36800 

35800 

35225 

34650 

34650 

low 

35025 

36425 

36800 

35800 

35225 

34650 

34650 

TSC 

high 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

low 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

HSDG 

high 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

low 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

Rel  Pay 

high 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

low 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

QMA 

high 

1883304 

1883304 

1883304 

1883304 

1883304 

1883304 

1 883304 

low 

1883304 

1883304 

1883304 

1883304 

1883304 

1883304 

1 883304 
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B.  PARAMETER  INPUTS  FOR  FIGURE  46 


Where  “Recruiters”  is  the  only  variable  that  is  fixed.  EB,  NCF,  and 
advertising  are  floated. 


2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCF 

high 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

Low 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

UE 

high 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

Low 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

Recruiters 

high 

3500 

3500 

3500 

3500 

3500 

3500 

3500 

Low 

2500 

2500 

2500 

2500 

2500 

2500 

2500 

LRP 

high 

7.44 

11.22 

11.28 

11.38 

11.43 

11.46 

11.67 

Low 

7.44 

11.22 

11.28 

11.38 

11.43 

11.46 

11.67 

Advertising 

high 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

Low 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

EB 

high 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

Low 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

NCO 

high 

35025 

36425 

36800 

35800 

35225 

34650 

34650 

Low 

35025 

36425 

36800 

35800 

35225 

34650 

34650 

TSC 

high 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

Low 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

HSDG 

high 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

Low 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

Rel  Pay 

high 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

Low 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

QMA 

high 

1883304 

1883304 

1883304 

1883304 

1883304 

1883304 

1 883304 

Low 

1883304 

1883304 

1883304 

1883304 

1883304 

1883304 

1 883304 
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C.  PARAMETER  INPUTS  FOR  TEST  CASE  1 

Where  “Recruiters”  is  the  only  variable  that  is  fixed.  EB,  NCF,  and 
advertising  are  floated. 


2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCF 

high 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

Low 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

UE 

high 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

Low 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

Recruiters 

high 

3500 

3500 

3500 

3500 

3500 

3500 

3500 

Low 

2500 

2500 

2500 

2500 

2500 

2500 

2500 

LRP 

high 

7.44 

11.22 

11.28 

11.38 

11.43 

11.46 

11.67 

Low 

7.44 

11.22 

11.28 

11.38 

11.43 

11.46 

11.67 

Advertising 

high 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

Low 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

EB 

high 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

Low 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

NCO 

high 

40000 

40000 

40000 

40000 

40000 

40000 

40000 

Low 

30000 

30000 

30000 

30000 

30000 

30000 

30000 

TSC 

high 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

Low 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

HSDG 

high 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

Low 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

Rel  Pay 

high 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

Low 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

QMA 

high 

1883304 

1883304 

1883304 

1883304 

1883304 

1883304 

1 883304 

Low 

1883304 

1883304 

1883304 

1883304 

1883304 

1883304 

1 883304 

193 


D.  PARAMETER  INPUTS  FOR  TEST  CASE  2 

Where  “Recruiters”  is  the  only  variable  that  is  fixed.  EB,  NCF,  and 
advertising  are  floated. 


2015 

2016 

2017 

2018 

2019 

2020 

2021 

NCF 

high 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

Low 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

UE 

high 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

0.08 

Low 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

0.04 

Recruiters 

high 

3500 

3500 

3500 

3500 

3500 

3500 

3500 

Low 

2500 

2500 

2500 

2500 

2500 

2500 

2500 

LRP 

high 

7.44 

11.22 

11.28 

11.38 

11.43 

11.46 

11.67 

Low 

7.44 

11.22 

11.28 

11.38 

11.43 

11.46 

11.67 

Advertising 

high 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

Low 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

34.8264 

EB 

high 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

Low 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

40.971 

NCO 

high 

40000 

40000 

40000 

40000 

40000 

40000 

40000 

Low 

30000 

30000 

30000 

30000 

30000 

30000 

30000 

TSC 

high 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

Low 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

0.7 

HSDG 

high 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

Low 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

Rel  Pay 

high 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

1.2 

Low 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

QMA 

high 

1883304 

1873304 

1863304 

1853304 

1 843304 

1833304 

1 823304 

Low 

1873304 

1863304 

1853304 

1 843304 

1833304 

1823304 

1813304 
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APPENDIX  E.  SCENARIO  OF  INTEREST  TO  N1 


The  original  baseline  scenario  request  for  a  PROM-WED  run  was: 

For  your  baseline  scenario  we  can  use  the 

•  Current  program  of  record  for  Recruiting  mission  is  about  34000,  so  use: 
30000  -  40000 

•  Advert:  $60M  so  use  $40M  -  $  1 00M 

•  EB:  $55M  -  80M  range 

•  Recruiters  (use  Current  onboard)  I  think  they  are  at  about  2900  so  use 
2500  -  3500 

•  Unemployment  rate  we  use  national  rate  and  forecast  per  the  Blue  Chip 
Economic  Indicators  long  range  forecast.  Which  has  current  UE  at  -5.0% 
so  use  (4.0%  -  8.0%) 

•  Vary  relative  pay  between  .8  and  1.2 

(Palmer,  personal  communication,  14  Sep  2016) _ 


Following  continued  communication  with  N1,  the  baseline  scenario 
transitioned  into  a  best  case,  worst  case,  and  most  likely  case  exploration.  The 
following  scenarios  originated  from  that  request.  Test  Case  1  and  2,  explored 
within  the  report,  combines  all  three  of  these  cases  into  one  PROM-WED  run. 

A.  BEST  CASE 

The  Navy’s  best  case  scenario  would  be  a  low  recruiting  mission,  no 
limitation  on  the  number  of  recruiters  in  the  field,  and  favorable  economic 
conditions  for  recruiting  (i.e.,  high  unemployment  rate  and  relative  pay  favoring 
the  military  versus  the  civilian  sector).  Table  17  shows  the  variables  that  this 
scenario  focuses  on.  In  this  case,  all  decision  variables  will  be  optimized. 


Table  17.  Scenario  of  Interest:  Best  Case 


Variable  Type 

Variable  Name 

Value  Low 

Value  High 

Decision  Variable 

Recruiters 

Float 

Market  Factor 

Recruiting  Mission  (NCO) 

30,00( 

D  recruits 

Market  Factor 

Relative  Pay 

1.00 

1.20 

Market  Factor 

Unemployment  Rate 

5.5% 

8.0% 

195 


B.  WORST  CASE 


The  Navy’s  worst  case  scenario  would  be  a  high  recruiting  mission,  a 
limited  number  of  recruiters  in  the  field,  and  an  economic  environment  that  is 
unfavorable  to  recruiting  (i.e.,  the  unemployment  rate  is  low  and  the  relative  pay 
favors  the  civilian  sector).  The  inputs  for  the  worst  case  scenario  are  shown  in 
Table  18.  In  this  case,  the  number  of  recruiters  is  fixed  and  all  other  decision 
variables  will  be  optimized. 


Table  18.  Scenario  of  Interest:  Worst  Case 


Variable  Type 

Variable  Name 

Value  Low 

Value  High 

Decision  Variable 

Recruiters 

2,500  recruiters 

Market  Factor 

Recruiting  Mission  (NCO) 

40,000  recruits 

Market  Factor 

Relative  Pay 

0.80 

1.00 

Market  Factor 

Unemployment  Rate 

4.0% 

5.5% 

C.  MOST  LIKELY 

The  most  likely  scenario  that  the  Navy  will  face  is  a  moderate  recruiting 
mission,  a  limited  range  of  available  recruiters,  and  a  balanced  economic 
situation  that  naturally  fluctuates  between  favorable  and  unfavorable  conditions 
for  recruiting.  Table  19  shows  the  input  variables  for  this  scenario,  where  number 
of  recruiters  is  fixed  and  tested  over  a  range  of  values.  All  other  decision 
variables  will  be  optimized. 


Table  19.  Scenario  of  Interest:  Most  Likely 


Variable  Type 

Variable  Name 

Value  Low 

Value  High 

Decision  Variable 

Recruiters 

2,500  recruiters 

3,000 

recruiters 

Market  Factor 

Recruiting  Mission  (NCO) 

35,000  recruits 

Market  Factor 

Relative  Pay 

0.80 

1.20 

Market  Factor 

Unemployment  Rate 

4.5% 

6.5% 
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APPENDIX  F.  DISTRIBUTIONS 


A.  FY  201 7  DISTRIBUTIONS  FOR  TEST  CASE  1 


i  -  Distributions 


A  -  Advertising 


Hjoj - 1~  .. 

-i — — — — i — 1 — i — 1 — i — ' — i — i — i — ' — 

0  50  100  1  50  200  250  300  350 


21  -  EB 


Quantiles 

a  »  Summary  Statistics 

100.0% 

maximum 

323.6418 

Mean 

58.118597 

99.5% 

323.6418 

Std  Dev 

56.188302 

97.5% 

278.0274 

Std  Err  Mean 

4.9471041 

90.0% 

120.9422 

Upper  95%  Mean 

67.907288 

75.0% 

quartile 

73.9048 

Lower  95%  Mean 

48.329906 

50.0% 

median 

39.7348 

N 

129 

25.0% 

quartile 

22.92 

10.0% 

13.8271 

2.5% 

7.61005 

0.5% 

6.5819 

0.0% 

minimum 

6.5819 

Quantiles 

A  »  Summary  Statistics 

100.0% 

maximum 

427.2072 

Mean 

76.716542 

99.5% 

427.2072 

Std  Dev 

74.168552 

97.5% 

366.996125 

Std  Err  Mean 

6.5301769 

90.0% 

159.6437 

Upper  95%  Mean 

89.637612 

75.0% 

quartile 

97.5543 

Lower  95%  Mean 

63.795471 

50.0% 

median 

52.4499 

N 

129 

25.0% 

quartile 

30.2544 

10.0% 

18.2518 

2.5% 

10.0453 

0.5% 

8.6882 

0.0% 

minimum 

8.6882 
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B 


FY  2017  DISTRIBUTIONS  FOR  TEST  CASE  2 


■»  Distributions 


A  •»  Advertising 


H_L^J - 1 . 

1  1 — , — i  i  , — i — i  , — . 

— — — — — 1 — i — 1 — i — 1 — i — 1 — i — 1 — r 

0  50  100  150  200  250  300  350 


21  -  EB 


Quantiles 

A 

-  Summary  Statistics 

100.0%  maximum 

342.3389 

Mean 

71.101741 

99.5% 

342.3389 

Std  Dev 

60.307798 

97.5% 

278.834575 

Std  Err  Mean 

5.3098056 

90.0% 

131.319 

Upper  95%  Mean 

81.608099 

75.0%  quartile 

86.7565 

Lower  95%  Mean 

60.595384 

50.0%  median 

50.8824 

N 

129 

25.0%  quartile 

31.68145 

10.0% 

20.0803 

2.5% 

10.519725 

0.5% 

6.7331 

0.0%  minimum 

6.7331 

Quantiles 

A 

r  Summary  Statistics 

100.0%  maximum 

451.8873 

Mean 

93.854303 

99.5% 

451.8873 

Std  Dev 

79.606289 

97.5% 

368.061575 

Std  Err  Mean 

7.0089429 

90.0% 

173.341 

Upper  95%  Mean 

107.72269 

75.0%  quartile 

114.5186 

Lower  95%  Mean 

79.985912 

50.0%  median 

67.1648 

N 

129 

25.0%  quartile 

41.8195 

10.0% 

26.506 

2.5% 

13.88605 

0.5% 

8.8877 

0.0%  minimum 

8.8877 
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